Systems and methods for tracker characterization and verification

ABSTRACT

The present application relates to systems and methods used to characterize or verify the accuracy of a tracker comprising optically detectable features. The tracker may be used in spatial localization using an optical sensor. Characterization results in the calculation of a Tracker Definition that includes geometrical characteristics of the tracker. Verification results in an assessment of accuracy of a tracker against an existing Tracker Definition.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation of prior U.S. patentapplication Ser. No. 15/606,723, filed May 26, 2017 (now U.S. Pat. No.______), and entitled, “Systems and Methods for Tracker Characterizationand Verification” incorporated herein by reference and which priorapplication claims the benefit of the following four US provisionalapplications, all having the same title and all incorporated herein byreference: Provisional Patent Application No. 62/342,366, filed May 27,2016, entitled “Systems and Methods for Tracker Characterization andVerification”; Provisional Patent Application No. 62/349,887, filed Jun.14, 2016; Provisional Patent Application No. 62/354,355, filed Jun. 24,2016; and Provisional Patent Application No. 62/362,857, filed Jul. 15,2016.

FIELD

The present application relates to trackers having optically detectablefeatures for tracking objects in a space and more particularly tosystems, methods and devices to calculate a Tracker Definition throughcharacterization, or to verify the accuracy of a tracker with respect toa known Tracker Definition. Trackers may include those attached tobones, instruments or other objects in surgical localization systems.

BACKGROUND

Localization systems that use optical sensors to detect and measure alocation of an object in space may rely on the assumption that theoptically detectable features of an object (or of a tracker attached tothe object) are spatially located in a known pattern or geometry.However, there are several drawbacks to relying on this assumption. Thetracker or its optically detectable features may be damaged due tonormal wear and tear causing deviations of its physical geometry.Repeated sterilization to allow use in a sterilized environment may leadto deformities in physical geometry that may not be caught throughvisual inspection. Such inaccuracies may result in incorrect posecalculation during localization.

BRIEF SUMMARY

There is a desire to obtain information about the spatial location ofthe optically detectable features that are being tracked or verify theaccuracy of the features with respect to an expected geometry.

BRIEF DESCRIPTION OF DRAWINGS

Embodiments disclosed herein will be more fully understood from thedetailed description and the corresponding drawings, which form a partof this application, and in which:

FIG. 1 depicts a localization system using a camera attached to apatient's anatomy and a tracker attached to a surgical instrument inaccordance with the prior art;

FIG. 2 is a block diagram depicting how to detect a pose of a trackerusing an image feed of an optical camera and a Static TrackerDefinition;

FIG. 3 shows an example of a characterization jig with a plurality ofmounting locations;

FIG. 4 shows an embodiment of a localization system comprising a cameraand a tracker being used with a characterization jig of acharacterization system where the jig comprises mounting locations forthe camera and tracker;

FIGS. 5A and 5B depict two views (from the perspective of a camera) ofthe tracker attached to different tracker mounting locations on acharacterization jig comprising mounting locations configured to receivea tracker;

FIG. 6 shows a plurality of images from an optical feed of an opticalcamera;

FIGS. 7A and 7B are flowcharts showing workflow and other computeroperations to generate a Tracker Definition;

FIG. 8 shows a block diagram depicting a characterization operationproviding input to localization operations as an example for clarity;

FIGS. 9A and 9B are flowcharts showing workflow and other computeroperations to verify the accuracy of a tracker;

FIG. 10 shows a block diagram depicting a verification operation (partof characterization operations) that provides a Tracker AssessmentParameter to a localization operation as an example for clarity;

FIGS. 11A and 11B are flowcharts showing workflow and other computeroperations to calculate a Tracker Assessment Parameter;

FIG. 12 shows a block diagram depicting a verification operation (partof characterization operations) that provides a Tracker AssessmentParameter to a display unit as an example for clarity;

FIG. 13 depicts an exemplary characterization operation to show howinputs are processed in order to calculate a Tracker Definition;

FIG. 14 depicts two sides/components of a kinematic mount that usesthree pairs of attractive magnets and three pairs of pins and slots asan example for clarity;

FIG. 15 shows another configuration of a characterization jig with onemounting location for a tracker in a different plane as an example forclarity;

FIGS. 16A and 16B shows the components of a camera with a shroud and aclamp and the assembled shroud, camera and clamp assembly;

FIG. 17 shows calibration divots on another exemplary configuration of acharacterization jig;

FIG. 18 shows a probe with a tracker attached to it probing divots on acharacterization jig;

FIGS. 19A and 19B are flowcharts showing workflow and other computeroperations to calculate a Tracker Base Geometry;

FIG. 20 shows a characterization jig on a lid of a sterilization tray;

FIG. 21 illustrates a characterization system using a handheld cameraconnected to a computing unit to capture image data showing both atracker and a Golden Tracker separated by a fixed positionalrelationship;

FIG. 22A illustrates a configuration of the characterization system inwhich the tracker and the Golden Tracker are attached to acharacterization jig;

FIG. 22B illustrates a configuration of the characterization system inwhich the tracker, Golden Tracker and the camera are attached to thecharacterization jig;

FIG. 23 illustrates a configuration of the characterization system inwhich the tracker and the Golden Tracker are attached to each other;

FIGS. 24A and 24B depict a user method and a computer-implemented methodrespectively to calculate tracker definition using the Golden Tracker;

FIGS. 25A and 25B show block diagrams illustrating the inputs intocharacterization operations to determine a tracker definition and thetracker definition being used as an input into localization operationsrespectively;

FIG. 26 shows a characterization operation as an example for clarity;

FIG. 27 shows a block diagram illustrating the inputs into verificationoperations to assess a tracker;

FIGS. 28A and 28B depict a user method and a computer-implemented methodrespectively to verify the accuracy of a tracker with respect to aStatic Tracker Definition;

FIG. 29 shows a verification operation using one image as an example forclarity;

FIGS. 30A and 30B depict a characterisation jig as a vertical platehaving tracker mounting surfaces on opposite faces;

FIG. 31 illustrates a characterization jig with two tracker mountinglocations and a pivot leg, as an example for clarity;

FIG. 32 illustrates a system including the jig and an optical sensorconnected to a computing unit;

FIG. 33 illustrates the jig being articulated in a rotatory motion;

FIGS. 34A and 34B illustrate a method of use and a computer implementedmethod of use for calculating a Tracker Definition using image datacollected during articulation;

FIGS. 35A-35D illustrate examples of when a sensor is attached to a toolin different applications;

FIGS. 36A and 36B are block diagrams illustrating the inputs intocharacterization operations that determine a Tracker Definition andillustrating the output of a characterization operation being used as aninput in localization operations;

FIG. 37 shows a characterization operation as an example for clarity;

FIGS. 38A and 38B are block diagrams illustrating the inputs intocharacterization operations that assess the accuracy of a tracker andillustrating the output of a verification operation being used as aninput in localization operations;

FIGS. 39A-39C show the optical sensor, a shroud and a clamp asindividual components, as assembled, and as part of a characterizationsystem respectively;

FIGS. 40A and 40B illustrating two configurations of a system using asensor comprising a camera (optical sensor) and an accelerometer(inertial sensor);

FIGS. 41A and 41B illustrate a user method and a corresponding computerimplemented method to calculate a Tracker Definition;

FIG. 42 illustrates a block diagram showing the inputs into acharacterization operation to calculate a Tracker Definition;

FIG. 43 illustrates a block diagram depicting the Tracker Definitionbeing used in localization operations to calculate a pose of thetracker;

FIG. 44 illustrates a block diagram showing the inputs into averification operation to calculate a Tracker Assessment Parameter toverify the accuracy of a tracker with respect to a Static TrackerDefinition;

FIG. 45 illustrates a block diagram depicting the Tracker AssessmentParameter being used in localization operations to calculate a pose ofthe tracker; and

FIG. 46 depicts an exemplary characterization operation.

It will be appreciated that for simplicity and clarity of illustration,elements shown in the figures have not necessarily been drawn to scale.For example, the dimensions of some of the elements may be exaggeratedrelative to other elements for clarity.

DETAILED DESCRIPTION

Reference in the specification to “one embodiment”, “preferredembodiment”, “an embodiment”, or “embodiments” means that a particularfeature, structure, characteristic, or function described in connectionwith the embodiment is included in at least one embodiment, and may bein more than one embodiment. Also, such phrases in various places in thespecification are not necessarily all referring to the same embodimentor embodiments.

A computing unit may comprise a laptop, workstation, or other computingdevice having at least one processing unit and at least one storagedevice such as memory storing software (instructions and/or data) asfurther described herein to configure the execution of the computingunit.

A localization system provides spatial localization of objects(instruments, implants, anatomical structures) with respect to anotherobject such as a patient's anatomy. An example of a localization systemis described herein. A miniature localization system comprising a sensor(such as an optical sensor e.g.: a monocular camera) is used to detect atracker comprising optically detectable features of given shapes (e.g.,spheres) arranged in a particular pattern or constellation. Theoptically detectable features may be passive (e.g. constructed fromretroreflective materials) or active (e.g. LED light sources). Thetracker has a base that may be used to mount the tracker on the objectto be tracked. The base may comprise a kinematic mount that allows arepeatable connection between the tracker and the object. The systemfurther comprises a computing unit that is in communication with thecamera. The computing unit receives a real time feed of images from thecamera, extracts the optically detectable features (e.g., sphericalmarkers) and generates/calculates a six degrees of freedom (DOF) pose ofthe tracker in real-time. By knowing the spatial location of theoptically detectable features (relative to each other), the computingunit is able to calculate the position of the tracker, and hence theobject to which the tracker is attached. In order to calculate the poseof the tracker, one or more geometrical parameters related to themarkers, such as the pattern in which the markers are arranged, thedistance between two or more markers, the 3D geometry of the tracker,etc. are pre-loaded in to the computing unit. This pre-loadedinformation is called the Static Tracker Definition in thisspecification. This is the “expected” Tracker Definition: e.g. if thetracker is not physically deformed; if the optically detectable featuresare located where expected (by the computing unit); if the geometricaldefinition of the tracker conforms to mechanical specifications; etc.The Static Tracker Definition may be determined during the manufacturingprocess through adherence to strict tolerances described within themechanical specification of the tracker. Assuming that the StaticTracker Definition accurately represents the physical tracker during alocalization procedure, instructions executing on the computing unit canaccurately calculate the pose of the tracker.

However, there are several drawbacks to relying on the assumption thatthe Static Tracker Definition accurately represents the physicaltracker. If the tracker relies on individual markers being attachedthereto, the accuracy of the physical tracker geometry depends on themanufacturing quality of the markers, the accuracy of installation ofthe markers on the tracker, etc. In addition, the tracker may be damageddue to normal wear and tear causing deviations of its physical geometryfrom its Static Tracker Definition. Repeated sterilization to allow useof the tracker in a sterilized environment may lead to deformities inits physical geometry that may not be caught through visual inspection.Inaccuracies in the Static Tracker Definition may result in incorrectpose calculation in the localization procedure.

There is a desire to obtain a new Tracker Definition for a given trackerquickly and accurately and/or to verify the accuracy of a tracker withrespect to the Static Tracker Definition.

This specification describes two systems—one for localization and theother for characterization. It is to be understood that these may beunique systems that utilise the same hardware such as the camera tocapture images and the computing unit to execute instructions. Thesesystems may also be distinct units that use different hardware. It isalso possible that these discrete systems communicate with each other topresent a seamless user experience but optionally utilise the samehardware. Alternatively, the characterization system may be usedseparately from the localization system. For example, thecharacterization system may be used as part of quality control at amanufacturing site while manufacturing a localization system for use ina surgical setting. The hardware and software used to characterize thetracker may be completely distinct.

FIG. 1 illustrates a localization system 100 where a sensor comprisingan optical sensor (e.g.: camera) 102 communicates with a computing unit104. The terms “optical sensor” and “camera” are used interchangeably inthis specification. Images of a tracker 106 comprising opticallydetectable features (e.g. markers) 108 are captured by the opticalsensor 102. The pose (position and orientation) of the tracker 106 canbe calculated by the computing unit 104 and spatial measurements (basedon the pose of the tracker) may be displayed on a graphical userinterface (GUI) of the computing unit 104. The optical sensor 102 may bea monocular camera. A monocular camera can be small and compact. It maybe attached to a platform near the operating table on which the patientis located, to a tool or instrument used during surgery, or directly tothe patient's anatomy 110. The optical sensor 102 may be attached to theplatform through a kinematic mount mechanism. The kinematic mountmechanism may be on the camera 102 itself or on another holdingmechanism such as a clamp that is configured to hold a camera 102 inplace by restricting its movement. The computing unit 104 may use thepose of the target to further display relevant measurements. The tracker106 may be attached to an object of interest, a surgical instrument 112or to the anatomy of a patient 110.

A corresponding workflow 200 to calculate the pose of a tracker 106 inthe localization system described above is illustrated in FIG. 2. Thecomputing unit 104 receives a real time feed of images 202 from theoptical sensor 102, accesses the Static Tracker Definition 204 (e.g. inmemory 206) and performs Localization Operations 208 in order tocalculate the pose of a tracker with respect to the optical sensor 102.The pose of the tracker 210 may then be used to provide the surgeonreal-time intra-operative spatial measurements to aid the surgicalprocedure.

This specification discloses various embodiments of systems and methodsand components for tracker characterization and verification. FIGS. 1-2are relevant to all.

Tracker Characterization and Verification with Characterization Jig

Systems, methods and components for Tracker Characterization andVerification using Characterization Jig are described predominantly withreference to FIGS. 1-19.

There is disclosed a system comprising: a characterization jigcomprising: a base; and a plurality of mounting locations on the base,each mounting location configured to selectively receive an opticalsensor or a tracker; and a computer-readable storage device storinginstructions which, when executed on a computing unit, configure thecomputing unit to: receive a first image of the tracker such that thetracker appears in a field of view of the optical sensor when one of thetracker and the optical sensor is attached to a first of the pluralityof mounting locations on the characterization jig and another of thetracker and the optical sensor is attached to a platform; receive asecond image of the tracker such that the tracker appears in the fieldof view of the optical sensor when at least one of the tracker and theoptical sensor having been attached to the first mounting location isattached to a second of the plurality of mounting locations on thecharacterization jig and the other of the tracker and the optical sensoris attached to the platform; and calculating at least one of: a trackerdefinition using the first image, the second image, and a spatialrelationship between the first of the plurality of mounting locationsand the second of the plurality of mounting locations; and a trackerassessment parameter using the first image, the second image, thespatial relationship between the first of the plurality of mountinglocations and the second of the plurality of mounting locations, and astatic tracker definition.

The plurality of mounting locations on the characterization jig may beconfigured to receive optical sensors. The characterization jig mayprovide the platform for the tracker.

The plurality of mounting locations on the characterization jig may beconfigured to receive trackers. The characterization jig may provide theplatform for the optical sensor. The system may further comprise theoptical sensor, where the optical sensor is configured to attach to theplatform for the optical sensor on the characterization jig.

The characterization jig may comprise an identifier to identify thespatial relationship between at least the first and the second of theplurality of mounting locations of the characterization jig. Theidentifier may be readable by the optical sensor.

The characterization jig may be made entirely of sterilisable material.It may be configured as a part of a lid of a tray of medicalinstruments.

At least one of the first or the second of the plurality of mountinglocations on the characterization jig may be in a different plane thanthe other mounting locations.

User instructions may be displayed to a user on a display unit.

The computing unit may be configured to calculate the tracker assessmentparameter and provide it to a display unit.

The computing unit may be configured to calculate the tracker assessmentparameter and the system may further comprise a display unit to displaythe tracker assessment parameter.

At least one of the tracker definition and the tracker assessmentparameter may be provided for a localization procedure.

There is disclosed a computer-implemented method comprising: receiving,by at least one computing unit, a first image of a tracker appearing ina field of view of an optical sensor when one of the tracker and theoptical sensor is attached to a first mounting location on acharacterization jig comprising a base and a plurality of mountinglocations and another of the tracker and the optical sensor is attachedto a platform; receiving, by the at least one computing unit, a secondimage of the tracker when the one of the tracker and the optical sensorhaving been attached to the first mounting location is attached to asecond mounting location on the characterization jig such that thetracker appears in the field of view of the optical sensor and the otherof the tracker and the optical sensor is attached to the platform; andcalculating, by the at least one computing unit, at least one of: atracker definition using the first image, the second image, and aspatial relationship between the first mounting location and the secondmounting location; and a tracker assessment parameter using the firstimage, the second image, the spatial relationship between the firstmounting location and the second mounting location, and a static trackerdefinition.

When the method calculates the tracker definition the method may providespatial measurements based on a pose between the optical sensor and thetracker for intra-operative localization with respect to an anatomy of apatient.

There is provided a computer-implemented method comprising: receiving,by at least one computing unit, a first image of the tracker when thetracker is attached to a first of the plurality of mounting locations ona characterization jig comprising a plurality of mounting locations whenthe tracker is appearing in the field of view of the optical sensorattached to a platform; receiving, by the at least one computing unit, asecond image of the tracker when the tracker is attached to a second ofthe plurality of mounting locations on the characterization jig when thetracker is appearing in the field of view of the optical sensor attachedto the platform; and calculating, by the at least one computing unit, atracker base geometry using the first image, the second image and asensor tracker mount spatial relationship.

There is disclosed a computer-implemented method comprising:calculating, by at least one computing unit, a sensor tracker mountspatial relationship between an optical sensor and a characterizationjig using additional features of the characterization jig, thecharacterization jig comprising a base and a plurality of mountinglocations; receiving, by the at least one computing unit, a first imageof the tracker when the tracker is attached to a first of the pluralityof mounting locations on the characterization jig when the tracker isappearing in the field of view of the optical sensor attached to theplatform; receiving, by the at least one computing unit, a second imageof the tracker when the tracker is attached to a second of the pluralityof mounting locations on the characterization jig when the tracker isappearing in the field of view of the optical sensor attached to theplatform; and calculating, by the at least one computing unit, a trackerbase geometry using the first image, the second image and the sensortracker mount spatial relationship.

There is disclosed a system comprising: a characterization jig: a base;a plurality of mounting locations; and at least three divots; and acomputer-readable storage medium storing instructions which, whenexecuted on a computing unit, configure the computing unit to: receivedivot image data comprising at least three images of a tracker attachedto a calibrated probe while the tracker is appearing in a field of viewof an optical sensor, the optical sensor attached to a platform, whenthe probe with the tracker attached to it is placed in each of at leastthree of the divots on the characterization jig; calculate a sensortracker mount spatial relationship using the divot image data and adivot tracker mount spatial relationship; receive a first image of thetracker when the tracker is attached to a first of the plurality ofmounting locations on the characterization jig when the tracker isappearing in the field of view of the optical sensor attached to theplatform; receive a second image of the tracker when the tracker isattached to a second of the plurality of mounting locations on thecharacterization jig and when the tracker is appearing in the field ofview of the optical sensor attached to the platform; and calculate atracker base geometry using the first image, the second image and thesensor tracker mount spatial relationship.

The computing unit may be configured to calculate a tracker definition.

The computing unit may be configured to, before capturing the divotimage data, calibrate a tip of the probe by pivoting the probe, havingan attached tracker, about a fixed point while the optical sensorcaptures images of the tracker appearing in the field of view, andcalculate the tip of the probe by executing center of rotationoperations.

The computing unit may be further configured to include the tracker basegeometry to a tracker definition.

There is disclosed a computer-implemented method comprising: receiving,by at least one computing unit, divot image data comprising at leastthree images of a tracker attached to a calibrated probe while thetracker is appearing in a field of view of an optical sensor, theoptical sensor attached to a platform, when the probe with the trackerattached to it is placed in each of at least three divots on acharacterization jig comprising a base and a plurality of mountinglocations; calculating, by the at least one computing unit, a sensortracker mount spatial relationship using the divot image data and adivot tracker mount spatial relationship; receiving, by the at least onecomputing unit, a first image of the tracker when the tracker isattached to a first of the plurality of mounting locations on thecharacterization jig when the tracker is appearing in the field of viewof the optical sensor attached to the platform; receiving, by the atleast one computing unit, a second image of the tracker when the trackeris attached to a second of the plurality of mounting locations on thecharacterization jig when the tracker is appearing in the field of viewof the optical sensor attached to the platform; and calculating, by theat least one computing unit, a tracker base geometry using the firstimage, the second image and the sensor tracker mount spatialrelationship.

The method may further comprise, before capturing divot image data,calibrating a tip of the probe by pivoting the probe, having an attachedtracker, about a fixed point while the optical sensor captures images ofthe tracker appearing in the field of view, and the computing unitfurther configured to calculate the tip of the probe by executing centerof rotation operations. The method may include calculating a trackerdefinition. The method may include including the tracker base geometryto a tracker definition.

There is disclosed a system comprising: a characterization jig: a base;a plurality of mounting locations; and at least three or more divots;and a computer-readable storage medium storing instructions which, whenexecuted on a computing unit, configure the computing unit to: receivedivot image data comprising at least three images of a tracker attachedto a platform while the tracker is appearing in a field of view of anoptical sensor, the optical sensor attached to a calibrated probe, whenthe probe with the optical sensor attached to it is placed in each of atleast three of the plurality of divots on the characterization jig;calculate a tracker sensor mount spatial relationship using the divotimage data and a divot sensor mount spatial relationship; receive afirst image of the tracker when the optical sensor is attached to afirst of the plurality of mounting locations on the characterization jigwhen the tracker attached to the platform is appearing in the field ofview of the optical sensor; receive a second image of the tracker whenthe optical sensor is attached to a second of the plurality of mountinglocations on the characterization jig when the tracker attached to theplatform is appearing in the field of view of the optical sensor; andcalculate a sensor mount spatial relationship using the first image, thesecond image and the tracker sensor mount spatial relationship.

There is disclosed a computer-implemented method to characterize aplurality of trackers simultaneously comprising: receiving, by at leastone computing unit, a first image of the plurality of trackers when thetrackers are in a fixed positional relationship to one another andappear simultaneously in a field of view of an optical sensor, theoptical sensor attached to a first of a plurality of sensor mountinglocations on a characterization jig, the characterization jig furthercomprising a base; and receiving, by at least one computing unit, asecond image of the plurality of trackers when the trackers are in thefixed positional relationship to one another and appear simultaneouslyin the field of view of the optical sensor, the optical sensor attachedto a second of the plurality of sensor mounting locations on thecharacterization jig; and calculating, by the at least one computingunit, a Tracker Definition for each tracker using the first image, thesecond image, information to distinguish the plurality of trackers and aspatial relationship between the first and the second of the pluralityof sensor mounting locations on the characterization jig.

There is disclosed a system comprising: a characterization jigcomprising: a base; and a plurality of mounting locations on the base;and a computer-readable storage medium storing instructions which, whenexecuted on a computing unit, configure the computing unit to: receive afirst image of a plurality of trackers when the trackers are in a fixedpositional relationship to one another and appear simultaneously in afield of view of an optical sensor, the optical sensor attached to afirst of the plurality of sensor mounting locations on thecharacterization jig; receive a second image of the plurality oftrackers when the trackers are in the fixed positional relationship toone another and appear simultaneously in the field of view of theoptical sensor, the optical sensor attached to a second of the pluralityof sensor mounting locations on the characterization jig; and calculatea tracker definition for each tracker using the first image, the secondimage, information to distinguish the plurality of trackers and aspatial relationship between the first and the second of the pluralityof sensor mounting locations on the characterization jig.

There is disclosed a computer-implemented method comprising: receiving,by at least one computing unit, a first image of a tracker appearing ina field of view of an optical sensor when the optical sensor is attachedto a platform and the tracker is attached to a first tracker mountinglocation on a characterization jig, the characterization jig comprisinga base and a plurality of tracker mounting locations; receiving a secondimage of the tracker when the tracker is attached to a second trackermounting location on the characterization jig such that the trackerappears in the field of view of the optical sensor; and calculating atleast one of: a tracker definition using the first image, the secondimage, and an inter tracker mount spatial relationship; and a trackerassessment parameter using the first image, the second image, the intertracker mount spatial relationship, and a static tracker definition.

There is disclosed a system comprising: a characterization jigcomprising: a base; and a plurality of tracker mounting locations on thebase, each tracker mounting location configured to receive a tracker;and a computer-readable storage device storing instructions which, whenexecuted on a computing unit, configure the computing unit to: receive afirst image of a tracker when the tracker appears in a field of view ofan optical sensor, the optical sensor attached to a platform and thetracker is attached to a first tracker mounting location on thecharacterization jig; receive a second image of the tracker when thetracker is attached to a second tracker mounting location on thecharacterization jig when the tracker appears in the field of view ofthe optical sensor; calculate at least one of a tracker definition usingthe first image, the second image, and an inter tracker mount spatialrelationship; and a tracker assessment parameter using the first image,the second image, the inter tracker mount spatial relationship, and astatic tracker definition.

There is disclosed a computer-implemented method comprising: receiving afirst image of a tracker when the tracker appears in a field of view ofan optical sensor, the tracker attached to a platform and the opticalsensor attached to a first sensor mounting location on acharacterization jig, the characterization jig comprising a base and aplurality of sensor mounting locations; receiving a second image of thetracker when the sensor is attached to a second sensor mounting locationon the characterization jig when the tracker appears in a field of viewof the optical sensor; and calculating at least one of: a trackerdefinition using the first image, the second image, and an inter sensormount spatial relationship; and a tracker assessment parameter using thefirst image, the second image, the inter sensor mount spatialrelationship, and a static tracker definition.

There is disclosed a system comprising: a characterization jigcomprising: a base; and a plurality of sensor mounting locations on thebase, each sensor mounting location configured to receive an opticalsensor; and a computer-readable storage device storing instructionswhich, when executed on a computing unit, configure the computing unitto: receive a first image of a tracker when the tracker appears in afield of view of an optical sensor, the tracker attached to a platformand the optical sensor attached to a first sensor mounting location onthe characterization jig; receive a second image of the tracker when thetracker appears in the field of view of the optical sensor, the trackerattached to the platform and the optical sensor attached to a sensorsecond mounting location on the characterization jig; and calculate atleast one of a tracker definition using the first image, the secondimage, and an inter sensor mount spatial relationship; and a trackerassessment parameter using the first image, the second image, the intersensor mount spatial relationship, and a static tracker definition.

It is understood that in any system described herein that includes acomputer storage medium additional components may be present such as acomputing unit, an optical sensor, and one or more targets, etc.

A characterization system that comprises a characterization jig may beused with the components of a localization system such as a sensor(e.g., a camera) and a tracker with optically detectable features (e.g.,markers). The camera may be the same hardware used for localization ormay be a part of a computing unit or laptop e.g., a webcam on a laptop.The characterization jig may be used to characterize a tracker (i.e.compute a current Tracker Definition) or to verify the accuracy of atracker with respect to a previously defined Tracker Definition. In oneexample, the characterization jig comprises at least two mountinglocations upon which a tracker can be attached. Two objects are“attached” when both are in contact with each other to form a connectionand there is a holding mechanism to enforce the connection. Theattachment may be rigid and removable, for example, selectivelyremovable. It may also be repeatable by virtue of the type of mechanismused for attachment, for e.g. a kinematic mount. The jig may be made ofsterilisable material to allow use within an operating room or any suchsterilized environment. The jig may optionally comprise a mountinglocation for a camera such that the camera can obtain a pose measurementof the tracker while the tracker is attached to any of the trackermounting locations. Alternatively, the camera may be attached to aplatform that is optionally a part of the characterization jig. Thegeometry of the characterization jig is known. This implies that thephysical shape, size, features etc. of the characterization jig areknown since the jig may be manufactured under strict tolerances. Theexamples used in this specification utilize a known spatial relationshipbetween the mounting locations of the tracker and therefore, between thecorresponding poses of the tracker. This known spatial relationship orany of the other geometrical characteristics may be pre-loaded into thememory of the computing unit of the characterization system (e.g. as ahard-coded value, as a 3D model, etc.) and used in the calculation ofthe Tracker Definition or the verification of the accuracy of a tracker.

This specification repeatedly references the spatial relationshipbetween multiple components, and uses the following nomenclature. Thedifference in position and orientation (in up to 6 degrees of freedom)of two components is referred to the spatial relationship betweenComponent_1 and Component_2, and labelled as Component_1 Component_2Spatial Relationship. The specification also refers to the spatialrelationship between multiple instances of the same component as InterComponent Spatial Relationship.

To verify the accuracy of a tracker, the camera captures imagesspecifically for characterization (also called “image data” in thisspecification) of the tracker when the tracker is attached to thetracker mounting locations on the characterization jig. Using a knownspatial characteristic of the characterization jig (such as an InterTracker Mount Spatial Relationship between the tracker mountinglocations), the computing unit compares image data of the tracker whenthe tracker is attached to each mounting location with image data thatis calculated using the pre-loaded Static Tracker Definition. Thecomputing unit may also calculate a new Tracker Definition and compareit with the Static Tracker Definition. This comparison results in adetermination of accuracy of the tracker. Alternatively, the new TrackerDefinition may be used as an input in a localization procedure. Thecharacterization and verification processes are further described below.

Reference is now made to FIG. 3 showing a characterization jig 300comprising a rigid base 302 and at least two mounting locations 304 fora tracker (e.g. 106). The characterization jig 300 may optionallycomprise a platform for a camera or a camera mounting location 306. FIG.4 shows a use of an embodiment of a characterization jig 300 along witha localization system 400 and a tracker 106. The camera 102 is incommunication with the computing unit 104. An exemplary characterizationjig 300 shown in this figure depicts a mounting location 306 on thecharacterization jig 300 for a camera 102 as well as three mountinglocations 304 on the same base for tracker 106 with optically detectablefeatures 108.

The Inter Tracker Mount Spatial Relationship may be stored in memory 206as a numerical representation of the relative poses between the mountinglocations represented as rotation matrices, quaternions, euler angles,translational vectors, Cartesian distances etc. The characterization jig300 may be manufactured to meet a specific Inter Tracker Mount SpatialRelationship, and the same spatial relationship may be accessible to thecomputing unit 104. For example, the numerical representation of thisspatial relationship may be data stored in memory 206, preferably in amanner that prevents or reduces the possibility of deletion. There maybe an identifier (not shown) such as a barcode, QR-code, URL, etc. onthe characterization jig 300 itself that identifies the Inter TrackerMount Spatial Relationship. The identifier may be read using the opticalcamera 102 and used to look up the Inter Tracker Mount SpatialRelationship that is stored in memory 206 or remotely such as at aserver on a network.

In an exemplary application of the disclosed characterization system,when optical sensor 102 is attached to its mounting location 306 and itsfield of view is directed towards tracker 106 attached to a trackermounting location 304 on characterization jig 300, optical sensor 102 isable to capture image data 600 (See FIG. 6) of tracker 106 (including atleast optically detectable features 108). Multiple mounting locations304 are provided and the Inter Tracker Mount Spatial Relationshipbetween tracker mounting locations 304 is known to computing unit 104that is in communication with camera 102. As the number of trackermounting locations 304 increases, a person skilled in the art willappreciate that the level of accuracy of the resulting calculations mayimprove. Tracker mounting locations 304 provide multiple vantage pointsto camera 102 such that the view of tracker 106 (as seen from the cameraattached to its mounting location) is substantially different in eachlocation. Computing unit 104 executes instructions to receive at leastone image from camera 102 while tracker 106 is in each mounting location304 to identify optically detectable features 108 (e.g. markers) in theimage data 600. Using the known Inter Tracker Mount Spatial Relationshipbetween each tracker mounting location 304 on characterization jig 300,computing unit 104 generates a Tracker Definition for tracker 106. Thismay be done by performing an optimization operation using the InterTracker Mount Spatial Relationship as an optimization constraint. Aperson skilled in the art will appreciate that a larger quantity ofimage data 600 from the camera may be acquired in order to improvecalculations in terms of speed, accuracy, robustness to errors(especially errors caused by a user), etc. The new Tracker Definitionmay then be provided to a localization procedure e.g., a navigatedsurgical procedure.

FIGS. 5A and 5B illustrate two views 500A and 500B of jig 300 havingtracker 106 mounted in two different tracker mounting locations 304similar to the perspective of camera 102. Note that the pose of thetracker is substantially different in each tracker mounting location 304and characterization jig 300 does not provide the platform for thecamera 102. This provides computing unit 104 with image data 600 fromvarious vantage points. Reference is now made to FIG. 6 which depictsmultiple instances of image data 600 comprising images of markers 108 oftracker 106 within a field of view of camera 102. Each instance of imagedata 600 corresponds to tracker 106 being attached at a differenttracker mounting location 304.

Reference is now made to flowcharts FIGS. 7A and 7B showing workflow andother computer operations. A workflow 700A executing on a computing unit104 directs a user to capture image (image data 600). At 702, operationsby computing unit 104 direct the user to set up the characterizationsystem by attaching camera 102 to sensor mounting location 306. The useris instructed (at 704) by workflow 700A (e.g. via a graphical userinterface and/or audible instructions or otherwise (not shown)) toattach tracker 106 on a first mounting location 304 and to initiate thecapture of image data 600. The user is then instructed (at 706) toattach tracker 106 to a second mounting location 304 and to initiate thecapture image data 600 for characterization. The user may be instructedto initiate the capture of data or the data may be capturedautomatically using methods like stability detection described furtherbelow. In a corresponding computer-implemented method 700B, computingunit 104 receives (at 708 and 710) image data 600 from camera 102(attached to optical sensor mounting location 306) of tracker 106 wheretracker 106 is sequentially attached to at least two tracker mountinglocations. Computing unit 104 may automatically capture image data 600via stability detection, or may receive it manually via a human machineinterface, such as a button on camera 102 or a command through computingunit 104 or foot pedal. In order to implement stability detection,computing unit 104 may capture image data 600 when optically detectablefeatures 108 in image data 600 appear to be stable for a substantialperiod of time (as appropriate for the application). Using image data600 and a known spatial relationship, such as the Inter Tracker MountSpatial Relationship between the various tracker mounting locations 304,computing unit 104 calculates a Tracker Definition 712. This TrackerDefinition can then be used in localization procedures to calculate pose210 of tracker 106. It is understood that optical sensor 102 and tracker106 locations may be reversed such that optical sensor 102 is movedamong a plurality of sensor mounting locations while tracker 106 remainsattached to a single tracker mounting location 304.

The instructions executing on computing unit 104 preferably includeerror checks to ensure that the tracker 106 is correctly attached to theappropriate mounting locations 304 during the workflow. This error checkmay come in the form of an error residual of an optimization operationthat uses image data 600 as an input. If the error residual is too high,computing unit 104 may prompt the user to verify the accuracy of theposition of tracker 106 on tracker mounting location 304. Computing unit104 may further provide the Tracker Definition to a localization system.

FIG. 8 shows a block diagram depicting a characterization operationproviding input to localization operations. Computing unit 104 receivesimage data 600 from optical sensor 102 of tracker 106 attached tocharacterization jig 600 and receives from memory 206 a known spatialrelationship, such as Inter Tracker Mount Spatial Relationship 802.Computing unit 104 performs characterization operations 804 using imagedata 600 and known spatial relationship 802 to determine a trackerdefinition 806. Computing unit 104 then receives a real time feed ofimages 202 from optical sensor 102 and tracker definition 806 andperforms Localization Operations 808 in order to calculate the pose 210of a tracker with respect to optical sensor 102.

Tracker Verification

During tracker verification, computing unit 104 verifies the accuracy ofa tracker with respect to a Static Tracker Definition 204. Reference isnow made to FIGS. 9A and 9B which are flowcharts showing workflow andother computer operations 900A and 900B executing on computing unit 104.At 902 operations direct (e.g. via a GUI or other human machineinterface) a user to set up the characterization system by attachingcamera 102 to a sensor mounting location 306. The user is furtherinstructed by the workflow at 904 and 906 to attach tracker 106consecutively on a first mounting location 304 and a second mountinglocation 304 in order to capture image data 600 for verification at twolocations. It is understood that more captures at more locations may beinstructed and performed. A corresponding computer-implemented method900B is illustrated in FIG. 9B comprising steps 908, 910 and 912.Computing unit 104 captures image data 600 from camera 102 attached to asensor mounting location 306 of tracker 106 attached to a first trackermounting location 304, and a second tracker mounting location 304. Usinga known Inter Tracker Mount Spatial Relationship between trackermounting locations 304, computing unit 104 verifies the accuracy of atracker. This may be done in several ways such as by analyzing imagedata 600 associated with tracker 106 for consistency with a given StaticTracker Definition 204 (e.g., expected image data 600 using the StaticTracker Definition 204 may be generated by computing unit 104, andcompared with actual image data 600 captured by the camera 102), bycalculating a Tracker Definition 806 for tracker 106 and comparing theresults with Static Tracker Definition 204, etc.

It is possible that for some surgical applications, deviation of tracker106 from a Static Tracker Definition 204 is unacceptable. The system maygenerate a Tracker Assessment Parameter to quantify this deviation. ThisTracker Assessment Parameter may then be used by computing unit 104during a localization procedure to determine whether the tracker 106being used is accurate or not. Some of the examples of how the TrackerAssessment Parameter is presented could be a Boolean flag, a singlenumerical value or multiple numerical values associated with how closelyTracker Definition 806 matches Static Tracker Definition 204, etc. FIG.10 shows a block diagram depicting a verification operation 1000 (partof characterization operations) that provides a Tracker AssessmentParameter 1006 to a localization operation 1008. The localization systemfurther calculates a pose 210 of the tracker 210. As describedpreviously, the characterization operations 804 and localizationoperations 1008 may be executed in the form of instructions on the samecomputing unit 104.

Reference is now made to flowcharts FIGS. 11A and 11B showing workflow1100A and other computer operations 1100B to calculate a TrackerAssessment Parameter.

At 1102, workflow 1100A, executing on a computing unit 104, providesuser instructions to set up the system by attaching camera 102 to sensormounting location. At 1104 and then 1106 the user is further instructedby the workflow 1100A to attach tracker 106 consecutively on a firstmounting location 304 and a second mounting location 304 in order tocapture image data 600 for verification. Correspondingcomputer-implemented method 1100B calculate the Tracker AssessmentParameter 1006 in operations 1108, 1110 and 1112. Computing unit 104captures at 1108 and 1110 image data 600 of a tracker 106 attached to afirst tracker mounting location 304 and a second tracker mountinglocation 304 on a characterization jig 300. Using image data 600, knownInter Tracker Mount Spatial Relationship 802 between the trackermounting locations 304, and the Static Tracker Definition 204, thecomputing unit 104 calculates the Tracker Assessment Parameter 1006 at1112. The Tracker Assessment Parameter 1006 may simply be presented tothe user (e.g.: on a display unit 1206) to allow the user to makefurther decisions. FIG. 12 shows a block diagram depicting averification operation 1200 (part of characterization operations) thatprovides a Tracker Assessment Parameter to a display unit. Computingunit 104 receives image data 600 from the optical sensor 102 of thetracker 106 attached to the characterization jig 600 and receives frommemory 206 a known spatial relationship, such as the Inter Tracker MountSpatial Relationship 802 and the Static Tracker Definition 204. Thecomputing unit 104 performs characterization operations 1204 using theimage data 600, the known spatial relationship 802, and the StaticTracker Definition 204 to calculate the Tracker Accuracy AssessmentParameter 1006. The computing unit 104 then displays the TrackerAccuracy Assessment Parameter on a display unit 1206.

FIG. 13 depicts an exemplary characterization operation 1300 in a blockdiagram to show how inputs are processed in order to calculate TrackerDefinition 806. Characterization operation 1300 is shown in three stages1304. First, image data 600 (comprising multiple 2D images) receivedfrom camera 102 is image processed 1306 to discern the opticallydetectable features 108 (e.g., markers) in each instance of image data600. The locations of each optically detectable feature 1308, alsocalled the marker locations, in an image are used in the next stage 1310where the operations create a correspondence between an opticallydetectable feature of one image and the same feature in a second image,and so on. The resulting matched marker locations 1312 are optimized andpresented to the third stage 1314 where operations solve for a value forTracker Definition 806. Persons skilled in the art will understand thatthis is an exemplary operation, and that there may be a variety ofoperations that can be executed to perform such calculations.

Computing unit 104 may comprise multiple, distributed processing units(e.g. characterization may be carried out on one computing unit 104,whereas intra-operative localization may be carried out by anothercomputing unit 104). It will be evident to those skilled in the art whatdata must be transferred between various computing units 104 to enablethe functionality of the systems and methods described herein. Trackercharacterization or verification may be performed on an ad-hoc basis aspart of routine maintenance of trackers 106, or may be performed priorto each localization procedure that uses tracker 106. A person skilledin the art will appreciate that the level of accuracy of measurementswill improve with an increase in the number of mounting locations fortracker 304, or camera 306 and the corresponding image data 600 capturedby the camera 102.

Characterization jig 300 has a plurality of mounting locations that canreceive a sensor 306 or a tracker 304. In the exemplary configuration ofthe characterization jig 300 described above, multiple tracker mountinglocations 304 with a known Inter Tracker Mount Spatial Relationship 802are described. In another configuration, multiple camera mountinglocations 306 are provided and a known Inter Sensor Mount SpatialRelationship may be used. Tracker 106 may be attached to one trackermounting location 304 while the multiple camera mounting locations 306provide multiple vantage points of tracker 106. The instructions oncomputing unit 104 may be adapted accordingly. Each of the variationsdescribed above are applicable to a characterization jig 300 thatcomprises multiple mounting locations 306 for an optical sensor suc hascamera 102. An advantage of moving the camera 102 (instead of thetracker 106) to achieve the various vantage points is that the systemcan then characterize tracker 106 that does not have a repeatablemounting mechanism in its base. For example, the tracker 106 may be afully integrated probe with a tip and no mounting mechanism; theoptically detectable features 108 (e.g.: markers) may be attached to asurgical instrument directly instead of attaching tracker 106 to theinstrument, etc.

It may be desirable to characterize multiple trackers 106simultaneously. The characterization jig 300 may comprise at least twocamera mounting locations 306 for use with one camera 102 and a knownInter Sensor Mount Spatial Relationship. The multiple trackers 106 arefixed relative to each other i.e. there is a fixed positionalrelationship amongst the trackers 106. For example, tracker 106 may beattached via its own mount to another instance of a tracker 106, andmultiple trackers 106 may be attached to tracker mounting locations 304on a characterization jig 300. In another example, multiple trackers 106may be resting on a surface during characterization, etc. The multipletrackers 106 are within a field of view of the camera 102 when camera102 is attached to each of the at least two camera mounting locations306 on characterization jig 300. Image data 600 comprising at least twoimages of trackers 106 showing optically detectable features 108 of eachtracker 106 are provided to computing unit 104. Computing unit 104 mayhave access to additional information to distinguish between or amongmultiple trackers 106 in the image data 600. For example, thisinformation may be an approximate Tracker Definition (Rough TrackerDefinition) for each tracker 106, shapes of the optically detectablefeatures 108 etc. In another example, optically detectable features 108of each tracker 106 may be of different colours, and this difference maybe known. In another example, the system may be configured to havesubstantial spacing between or among trackers 106 during image capturesuch that the various trackers 106 can be distinguished within an imagebased on the spacing of optically detectable features 108 within imagedata 600. Computing unit 104 may calculate an instance of TrackerDefinition 806 for each tracker using image data 600, the Inter SensorMount Spatial Relationship and the additional information to distinguishbetween or among mulitple trackers 106.

Characterization jig 300 may be encoded with an optically readablebarcode or other identifier that can be used to load specificinformation about the spatial relationships between the various mountinglocations, comprising the tracker mounting locations 304 and the cameramounting locations 306. Each characterization jig 300 may have adifferent Inter Tracker Mount Spatial Relationship 802 or Inter SensorMount Spatial Relationship associated with it. At the beginning of thetracker verification or characterization process, camera 102 may readthe identifier to allow computing unit 104 to access known pre-loadedinformation, such as the spatial relationships between the variousmounting locations 802 for that characterization jig 300. It isunderstood that the information may be retrieved from a remote store(not shown) via a communication network (not shown). Similarly, tracker106 itself may be encoded with an optically readable barcode or otheridentifier or any other method of encoding that can be used to load ordistinguish its Static Tracker Definition 204 in computing unit 104. Theaccuracy of tracker 106 may then be verified with respect to its StaticTracker Definition 204.

A mounting location on characterization jig 300 can be any positionallyrepeatable mounting mechanism, preferably repeatable in six degrees offreedom. Depending on the application, it may be sufficient to use amount that is repeatable in less than six degrees of freedom as long asthe instructions executing on computing unit 104 take this factor intoaccount. In this case, it may be necessary to obtain more than twoimages from the camera. Any mechanism achieving positional repeatabilitymay form the repeatable mounting locations for tracker 106 and/oroptical sensor 102 on characterization jig 300. For example, a magnetickinematic mount may comprise two mating sides. Each side may haveattractive pairs of magnets to hold the mechanism together, and preciseslots/hemispherical pins that mate with each other when the mechanism is“connected”, and provide a highly repeatable kinematic mount. Akinematic mount is a mechanical interface which is highly repeatable(i.e. between connect and disconnect cycles) in up to 6 degrees offreedom (3 degrees of freedom in orientation and 3 degrees of freedom intranslation). Two views of an examplary kinematic mount 1400 (1400A,1400B) are illustrated in FIG. 14. Three pairs of hemispeherical pins1402 and corresponding slots 1404 or V-grooves mate using three pairs ofattractive magnets 1406 to hold the first side of the kinematic mount1400A together with the second side of the kinematic mount 1400B andenforce a kinematic connection. It is important to note that there areseveral kinds of kinematic mounts, an example of which is provided inthis specification. This kinematic mount is included as an example forclarity and is not meant to limit the scope of the specification. Anyother repeatable mounting mechanism and/or feature may be used, e.g.dovetail connections, cam locks, v-grooves, rails, flat surfaces, balldetents, clamps, etc.

A minimum of two repeatable mounting locations are required to enableTracker Characterization and Verification as described above. Multiplerepeatable mounting locations may increase the accuracy, performance orrobustness of the characterization or verification process. Where morethan two mounting locations are used, it may be desirable for at leastone mounting location to be in a different plane than the first twomounting locations. In FIG. 15, a characterization jig 1500 comprises arigid base 1502, a camera mounting location 306 and three trackermounting locations 304. The first tracker mounting location 304 andsecond tracker mounting location 304 differ by one degree of freedom inthe pose of the tracker 106 when attached to these tracker mountinglocations 304, whereas a third tracker mounting location 304 furtherprovides a difference in a second degree of freedom. In other words, therotations between the three tracker mounting locations are about twodifferent 3D vectors.

In an exemplary configuration of the characterization jig 1500, a singlemounting location using a magnetic kinematic mount may provide multipleposes of the tracker 106 to the camera 102, such that the difference inrotation between each of the poses is known. The single mountinglocation may effectively act as two (or more) mounting locations.Additionally, for example, the kinematic mount of FIG. 14 may permitindexing functionality. In the absence of other constraints, the mountmay be mated in a total of three orientations, each spaced at 120degrees. In this case, in at least one of three mounting configurations,the tracker 106 is likely facing away from the camera 102, and may notgenerate a valid image for pose calculation due to optical occlusion.However, the remaining two mounting configurations may yield a vantagepoint from which a valid pose of the tracker 106 may be captured. Thedifference in rotation in these two mounting configurations is 120degrees. In the example depicted in FIG. 14, the number of slots 1404(or pins 1402) could be increased to provide finer indexing capability(e.g. doubling the number of equally spaced slots 1404 would providerotational indexing at 60 degree increments). Various types of indexingmounts are contemplated, including the use of indexing adaptors. Theadvantage of indexing mounts is that a single mount may effectivelyserve as multiple mounts, without the cost/complexity/size/footprintassociated with multiple mounts.

It may be advantageous to obtain information about the geometry of theoptically detectable features of the tracker with respect to the base ofthe tracker. This information, also known as the Tracker Base Geometry,may be added to the Tracker Definition and be used to allow alocalization system to determine the location of the object to which thetracker is attached. The steps to obtain Tracker Base Geometry aredescribed below. In order to obtain this information, the spatialrelationship between the camera and each tracker mounting location(referred to in this specification as Sensor Tracker Mount SpatialRelationship) is required. The base of the tracker may comprise arepeatable mounting mechanism. The camera mounting location and trackermounting locations may optionally be on the same characterization jig.

The camera may attach directly and repeatably to the characterizationjig or be attached with the use of a clamp or other means. Reference isnow made to FIG. 16A. A clamp assembly 1600 consists of an opticalsensor 102 and a clamp 1602, and optionally a shroud 1604. The shroud1604 fixes the optical sensor 102, and is placed within the clamp 1602.The shroud 1604 (shown in a simplified manner) has an outer surfacewhich matches a mating surface 1606 on the inside of the clamp 1604.Each of the mating surfaces may define portions of a sphere, such thatthe clamp 1602/shroud 1604 interface provides an alignment mechanismthat is functionally a lockable ball joint. The clamp 1602 has amechanism (e.g. a screw/hinge combination) which applies a force on theshroud 1604 (and, in turn, the optical sensor 102), and clamps itrigidly and releasably in place. The shroud 1604 and the clamp 1602 haverespective mating surfaces which, when the optical sensor 102 is in theshroud 1604 and the clamp 1602 is in a partially closed position, enablerelative movement of the shroud 1604 and clamp 1602 to adjust theorientation of the optical sensor 102. The clamp 1602 has a base 1608which may be a kinematic mount. FIG. 16B shows a camera 102 assembledwith a shroud 1604 and clamp 1602. Since the sensor 102, when fixedwithin the clamp assembly 1600 and attached to the sensor mountinglocation 306 on the characterization jig, is not in a known spatialrelationship with respect to the characterization jig, thecharacterization jig must be registered to the optical sensor 102 i.e.the optical sensor 102 must localize the characterization jig in space,as described further below.

Reference is now made to FIG. 17 showing an exemplary configuration of acharacterization jig 1700 comprising a rigid base 302, tracker mountinglocations 304, a camera mounting location 206, and additional featuressuch as calibration divots 1704. Once the optical sensor 102 is attachedto its mounting location 306 on the characterization jig 1700, the threecalibration divots 1702 are used to register the characterization jig1700 to the optical sensor 102. The three calibration divots 1702 arenot collinear since collinear divots would not yield enough informationto perform registration.

If a probe 1612 is used to localize calibration divots 1702 with respectto the camera 102, then the probe 1612 itself must be calibrated (i.e.the position of the tip of the probe with respect to the markers 108 onthe tracker 106 must be known). A different tracker 106 may be used forthis purpose. The probe 1612 and tracker 106 may be pre-calibrated i.e.the position of the tip may be known via the manufacturing and/orinspection process or determined via a calibration procedure. FIG. 18illustrates a characterization system 1800, a probe 1612 with a tracker106 attached to it is placed in one of the calibration divots 1702 onthe characterization jig 1700. An exemplary probe calibration processinvolves the following steps: the tracker 106 is attached to the probe1612 and is pivoted about a fixed point (e.g.: one of the calibrationdivots 1702 on the characterization jig 1700) while being tracked by thecamera 102. Using images of the tracker 106 generated during pivoting, acomputing unit 104 executes a “center of rotation” operation todetermine the location of the tip of the probe.

As described previously, the geometrical spatial relationships 802between the various features of the characterization jig 1700, its 3Dshape, etc. are known. For calculation of a Tracker Definition 806,instructions executing on the computing unit 104 utilize the knownspatial relationships between a first mounting location and a secondmounting location 802. In the examples above, the use of the InterTracker Mount Spatial Relationship 802 or the Inter Sensor Mount SpatialRelationship has been described. Similarly, the spatial locations ofeach of the three calibration divots with respect to the trackermounting locations (Divot Tracker Mount Spatial Relationship) is knownto the computing unit 104 and are used in the calculation of TrackerBase Geometry. FIGS. 19A and 19B are flowcharts showing workflow 1900Aand other computer operations 1900B to calculate a Tracker BaseGeometry. At 1902, a computing unit instructs a user to set up thesystem to calculate Tracker Base Geometry. The user is instructed, by aworkflow executing on the computing unit 104, to place the probe 1612(with the tracker 106 attached to it) in each of the three (or more)calibration features and initiate capture of divot image data 600 thatcomprises at least three images of the tracker (at 1904 and 1906). Acorresponding computer-implemented method is illustrated where acomputing unit 104 captures divot image data from a camera 102 with theprobe 1612 (with the tracker 106 attached to it) placed in eachcalibration divot 1702, 1908. The computing unit 104 then captures imagedata 600 from a camera 102 with the tracker 106 attached to each trackermounting location 304, 1910. The computing unit 104 utilizes the divotimage data, as well as the known Divot Tracker Mount SpatialRelationship to determine the Sensor Tracker Mount Spatial Relationship1912. The computing unit 104 further utilizes the Sensor Tracker MountSpatial Relationship, as well as image data 600 of a tracker 106attached to a particular tracker mounting location 304, to calculate theTracker Base Geometry 1914. The Tracker Base Geometry may be added tothe Tracker Definition 806. The Sensor Tracker Mount SpatialRelationship is required for determining the Tracker Base Geometry. TheSensor Tracker Mount Spatial Relationship may be determined using aknown spatial relationship between the optical sensor and the sensormounting location, and the Sensor Mount Tracker Mount SpatialRelationship. The computing unit 104 uses both known spatialrelationships to determine the Sensor Tracker Mount SpatialRelationship. To compute the Tracker Base Geometry, the computing unit104 further utilizes the Sensor Tracker Mount Spatial Relationship, aswell as image data 600 of a tracker 106 when attached to trackermounting locations 304.

If the positions of the optical sensor 102 and tracker 106 on thecharacterization jig 1700 are reversed, the resulting calculation isthat of a Sensor Sensor Mount Spatial Relationship, instead of a TrackerBase Geometry for the tracker.

In addition to using the calibration divots 1702 for determininggeometry of the markers 108 to the base of the tracker, the calibrationdivots 1702 may also be used to verify the accuracy of the measurementsprovided by the localization system. The computing unit 104 may beadapted to execute instructions such that a user can check if a physicallocation of the tip of the probe 1612 on a known geometrical model of acharacterization jig 1700 corresponds to the location obtained throughlocalization.

For use in surgical applications, the characterization jigs describedabove may be made entirely of sterilisable materials. Thecharacterization jigs may also be configured to fit within asterilization tray. It may be integrated into a lid of the sterilizationtray or be a separate component. FIG. 20 illustrates a lid of asterilization tray 2000 (typically used in surgical settings) configuredto be a characterization jig with a camera mounting location 306 andtracker mounting locations 304.

Tracker Characterization and Verification with Golden Tracker

Systems, methods and components for tracker characterization andverification using a Golden Tracker are predominantly described withreference to FIGS. 1-2, 6, and 20-30 though it will be understood thatthere is some overlap with concepts etc. to earlier or later describedembodiments herein. There is described and shown a computer-implementedmethod comprising the steps of: receiving, by a computing unit, a firstimage of a tracker and a Golden Tracker, separated by a fixed spatialrelationship, the first image captured by an optical sensor from a firstvantage point when both trackers appear in a field of view of theoptical sensor; receiving, by the computing unit, a second image of atracker and a Golden Tracker, separated by the fixed spatialrelationship, the second image captured by the optical sensor from asecond vantage point when both trackers appear in the field of view ofthe optical sensor; and calculating, by the computing unit, at least oneof: a Tracker Definition using the first image, the second image, animage difference, and the Golden Tracker Definition; and a TrackerAssessment Parameter using the first image, the second image, the GoldenTracker Definition and a Static Tracker Definition.

The method may further comprise calculating, by the computing unit, theimage difference by calculating a difference in pose of the GoldenTracker between the first image and the second image using the firstimage, the second image and the Golden Tracker Definition.

The fixed spatial relationship may be an Inter Tracker Mount SpatialRelationship of a characterization jig comprising mounting locations forthe tracker and Golden Tracker and when the method calculates theTracker Definition, the method may further comprise calculating, by thecomputing unit, a Tracker Base Geometry using the first image, thesecond image, the image difference, the Inter Tracker Mount SpatialRelationship, the Golden Tracker Definition and a Golden Tracker BaseGeometry.

The optical sensor may be attached to a platform and the tracker and theGolden Tracker are attached to each other.

When the method calculates the Tracker Definition, the method mayfurther comprise calculating, by the computing unit, a Tracker BaseGeometry using the first image, the second image, the image difference,the fixed spatial relationship, the Golden Tracker Definition and aGolden Tracker Base Geometry. The tracker and the Golden Tracker may beattached to each other.

The method may further comprise providing, by the computing unit, atleast one of the Tracker Definition or the Tracker Assessment Parameteras calculated for use in a localization procedure.

The method may further comprise providing, by the computing unit, userinstructions for display by a display unit.

When the method calculates the Tracker Assessment Parameter, the methodmay further comprise providing, by the computing unit, the TrackerAssessment Parameter for display by a display unit.

The tracker and the Golden Tracker may be attached to a respectivemounting location on a characterization jig.

There is shown and described a computer-implemented method comprisingthe steps of: receiving, by a computing unit, an image of a tracker anda Golden Tracker attached to each other from one vantage point when bothtrackers appear in a field of view of an optical sensor; calculating, bythe computing unit, a Tracker Assessment Parameter using the image, aGolden Tracker Definition, a Golden Tracker Base Geometry, a StaticTracker Definition and a Static Tracker Base Geometry.

There is shown and described a system comprising: a Golden Tracker; anda computer-readable storage device storing instructions and data,including a Golden Tracker Definition for the Golden Tracker, which,when executed on a computing unit, configure the computing unit to:receive a first image of a tracker and the Golden Tracker, separated bya fixed spatial relationship, the first image captured by an opticalsensor from a first vantage point when both trackers appear in a fieldof view of the optical sensor; receive a second image of the tracker andthe Golden Tracker, separated by the fixed spatial relationship, thesecond image captured by the optical sensor from a second vantage pointwhen both trackers appear in the field of view of the optical sensor;and calculate at least one of a Tracker Definition for the tracker usingthe first image, the second image, an image difference, and the GoldenTracker Definition; and a Tracker Assessment Parameter for the trackerusing the first image, the second image, the image difference, and aStatic Tracker Definition.

The computing unit may be further configured to calculate the imagedifference by calculating a difference in pose of the Golden Trackerbetween the first image and the second image using the first image, thesecond image and the Golden Tracker Definition.

The computing unit may be further configured to provide at least one ofthe Tracker Definition or the Tracker Assessment Parameter for use in alocalization procedure.

The computing unit may be further configured to provide userinstructions for display on a display unit.

The computing unit may be further configured to provide the TrackerAssessment Parameter when calculated for display on a display unit.

The system may further comprise a display unit to display the TrackerAssessment Parameter.

The tracker and the Golden Tracker may be attached to a respectivemounting location on a characterization jig.

The characterization jig may be made entirely of sterilisable materialor made entirely of non-sterilisable material.

The system may further comprise the characterization jig.

The system may further comprise the optical sensor.

The fixed spatial relationship may be an Inter Tracker Mount SpatialRelationship of a characterization jig comprising mounting locations forthe tracker and Golden Tracker and when the computing unit is configuredto calculate the Tracker Definition, the computing unit may be furtherconfigured to calculate a Tracker Base Geometry using the first image,the second image, the image difference, the Inter Tracker Mount SpatialRelationship, the Golden Tracker Definition and a Golden Tracker BaseGeometry.

The optical sensor may be attached to a platform and the tracker and theGolden Tracker are attached to each other.

When the computing unit is configured to calculate the TrackerDefinition, the computing unit may be further configured to calculate aTracker Base Geometry using the first image, the second image, the imagedifference, the fixed spatial relationship, the Golden TrackerDefinition and a Golden Tracker Base Geometry. The tracker and theGolden Tracker may be attached to each other.

There is shown and described a system for characterizing a tracker forlocalization in a surgical procedure, the system comprising: anon-sterile Golden Tracker; a sterile characterization jig comprising atleast two mounting locations and further configured to simultaneouslyattach the Golden Tracker and a sterile tracker on each mountinglocation respectively without compromising the sterility of the tracker;a computer-readable storage device storing instructions and data,including a Golden Tracker Definition for the Golden Tracker, which,when executed on a computing unit, configure the computing unit to:receive a first image of the tracker and the Golden Tracker captured byan optical sensor from a first vantage point when both trackers appearin a field of view of the optical sensor; receive a second image of thetracker and the Golden Tracker captured by the optical sensor from asecond vantage point when both trackers appear in the field of view ofthe optical sensor; and calculate at least one of: a Tracker Definitionfor the tracker using the first image, the second image, an imagedifference, and the Golden Tracker Definition; and a Tracker AssessmentParameter for the tracker using the first image, the second image, theimage difference, the Golden Tracker Definition and a Static TrackerDefinition.

This document describes two systems—one for localization and the otherfor characterization. It is to be understood that these may be distinctsystems that utilise the same hardware such as the camera to captureimages and the computing unit to execute instructions. These systems mayalso be distinct units that use different hardware. It is also possiblethat these distinct systems communicate with each other to present aseamless user experience but optionally utilise the same hardware.Alternatively, the characterization system may be used separately fromthe localization system. For example, the characterization system may beused as part of quality control at a manufacturing site whilemanufacturing a localization system for use in a surgical setting. Thehardware and software used to characterize the tracker may be completelydistinct.

In an embodiment, a tracker may be characterized i.e. its TrackerDefinition calculated by using a characterization system comprising a“Golden Tracker”, with a known Golden Tracker Definition, andinstructions executing on the computing unit. The “Golden Tracker”refers to a tracker whose optically detectable features have a spatialrelationship that precisely match its Golden Tracker Definition. Thedegree of precision may be dependent on the desired accuracy forlocalization measurements using the characterized tracker. In thisembodiment, the tracker and the Golden Tracker are separated by a fixedpositional relationship i.e. the relationship must not vary during theexecution of characterization and/or verification operations. An opticalsensor, in communication with a computing unit, is not required to beattached to a sensor mounting location and can be handheld to capture atleast one image each from multiple vantage points around the trackerssuch that the view of the trackers is substantially different from eachvantage point. Alternatively, the optical sensor may be attached tomultiple mounting locations on a characterization jig or to platformsconfigured to receive optical sensors; the mounting locations orplatforms thus providing the multiple vantage points. Or, the opticalsensor may be attached to a platform while the tracker and GoldenTracker are attached to each other and images are captured as bothtrackers are moved around together. Both trackers should appear in afield of view of the optical sensor from each vantage point. The imagedata comprising images of the tracker and Golden Tracker are provided bythe optical sensor as an input to characterization and/or verificationoperations executing on the computing unit. By identifying andclassifying the optically detectable features of the tracker and theGolden Tracker in the image data captured from various vantage pointsand the Golden Tracker Definition, instructions executing on thecomputing unit may calculate a Tracker Definition for the tracker (e.g.by performing an optimization routine). The accuracy of a tracker canalso be verified with respect to a previously defined TrackerDefinition. The verification process may further generate a TrackerAssessment Parameter. Exemplary operations are described further in thisdocument.

FIG. 21 illustrates a system 2100 of characterizing and/or verifyingtracker accuracy where the tracker 106 and Golden Tracker 2106 areseparated by a fixed relationship with respect to each other whileplaced on a flat surface 2102 e.g., a table. The optical sensor 102 isnot attached to a sensor mounting location 306 or a platform 2306 andcan be moved around to capture image data 2506 comprising at least twoimages of both trackers from two vantage points respectively. Theoptical sensor 102 is connected to a computing unit 104 that isexecuting instructions comprising characterization operations tocharacterize the tracker 106 and determine a Tracker Definition 806 orinstructions comprising verification operations 2702 to verify theaccuracy of the tracker 106 with respect to a known Static TrackerDefinition 204. The outputs of the characterization and/or verificationoperations may be fed into a localization system or displayed on adisplay unit 1206. In the exemplary figures of this document, bothtrackers appear to be of a similar geometrical shape and dimensions.This document also contemplates the use of a tracker 106 and GoldenTracker 2106 with different geometrical features. For example, thetrackers may have a different number of optically detectable features108, the spatial arrangement of the features may be different,geometrical characteristics of the optically detectable features mayvary, etc. These known differences can further be used, in addition tothe Golden Tracker Definition 2502, in the characterization and/orverification operations executing on the computing unit 104, forexample, to improve efficiency, speed or robustness of the operations.

A practical example of how a Golden Tracker 2106 may be used in asterile environment (such as an operating room) involves two nurses, oneof whom is a sterile nurse assisting the surgeon during a surgicalprocedure and can handle the sterile tracker 106 (that is part of alocalization system being used in the same operating room). The secondnurse, a non-sterile nurse, can handle the non-sterile Golden Tracker2106. Each nurse may respectively place the tracker 106 and GoldenTracker 2106 on a surface. The optical sensor 102 (that is optionally apart of the localization system) may be sterile or non-sterile; therespective nurse may move the optical sensor 102 such that it is aimedat both trackers, and capture image data. Since the spatial relationshipbetween the trackers is fixed (i.e. the trackers are not moved duringcharacterization or verification), operations executing on a computingunit 104 can determine a Tracker Definition 806 for the tracker 106 orverify its accuracy against a known Static Tracker Definition 204 usingthe difference in pose of the Golden Tracker 2106 from one vantage pointof the optical sensor 102 to another. An exemplary operation isdescribed in the paragraphs below.

The characterization system described in the example above mayoptionally comprise a characterization jig with at least two trackermounting locations to attach the tracker and Golden Trackerrespectively. Two objects are “attached” when both are in contact witheach other to form a connection and there is a holding mechanism toenforce the connection. The attachment may be rigid and removable, forexample, selectively removable. It may also be repeatable by virtue ofthe type of mechanism used for attachment, for example, a kinematicmount. The jig may be made entirely of sterilisable materials to allowuse in sterile environments. The optical sensor may be handheld orattached to additional sensor mounting locations on the jig or attachedto separate platforms to capture image data of the trackers while eachtracker is attached to a respective tracker mounting location. Theoptical sensor may capture at least one image from each vantage point toprovide input for the characterization operations. At least two vantagepoints must be used to capture the image data. With the trackers beingseparated by a fixed spatial relationship by virtue of the fixed trackermounting locations, the computing unit executes instructions to optimizevalues for the Tracker Definition using the Golden Tracker Definitionand image data.

Reference is now made to FIG. 22A of a system 2200A comprising acharacterization jig 2202 with at least two tracker mounting locations304 and a fixed spatial relationship between the tracker mountinglocations while the optical sensor 102 is handheld. One of the twotrackers is the Golden Tracker 2106 with a known Golden TrackerDefinition and the other is the tracker 106 with an unknown TrackerDefinition 806. The optical sensor 102 is in communication with acomputing unit 104 and is used to capture a plurality of images of thetrackers from a variety of vantage points or multiple viewing angles. Aperson skilled in the art will appreciate that for greater accuracy, alarger number of images may be captured during optimization. FIG. 22Billustrates a system 2200B comprising a characterization jig 2204 withmultiple tracker mounting locations 304 and sensor mounting locations306. The optical sensor 102 is moved from one sensor mounting location306 to another to capture image data 2506 for characterization orverification.

Reference is now made to FIG. 23 which illustrates a characterizationsystem 2300. System 2300 comprises Golden Tracker 2106, tracker 106,optical sensor 102, computing unit 104, and a platform 2306. GoldenTracker 2106 and tracker 106 are attached to one another. Optical sensor102 is coupled for communication with computing unit 104 and opticalsensor 102 is mounted (attached) to platform 2306. The tracker 106comprises a base 2302 and the Golden Tracker 2106 comprises a base 2304that allow it to be attached to another mount. Such attachment meansinclude magnets, springs, mating threads, cam-locks, etc. The attachmentmust be rigid. When using magnets as the attachment means, the magnetsof the tracker 106 may be complementary to allow a direct attachment tothe Golden Tracker 2106. Alternatively, an adapter may be used to allowan indirect attachment between the trackers. The adapter, with knowngeometrical dimensions, may have a first side comprising complementarymagnets to attach the tracker 106, and a second side comprisingcomplementary magnets to attach the Golden Tracker 2106. The use of theadapter may require the operations executing on the computing unit 104to be modified accordingly.

In the example of FIG. 23, the Golden Tracker Definition 2502 (see tooFIG. 25) includes the spatial relationship between the opticallydetectable features 108 and the base 2304 of the Golden Tracker. This isreferred to as the Golden Tracker Base Geometry 2920 (see too FIG. 29)in this document. When the tracker 106 and Golden Tracker 2106 areattached to each other, the optical sensor 102 may be attached to asensor mounting location 306 on a characterization jig or on a platform2306 and image data from multiple vantage points may be captured bymoving the tracker 106 and Golden Tracker 2106 instead of moving theoptical sensor 102. Using image data and the Golden Tracker Definitionincluding the Golden Tracker Base Geometry, the unknown TrackerDefinition 806 may be calculated. In this configuration, the resultingTracker Definition 806 also includes a Tracker Base Geometry for thetracker 106 i.e. the spatial relationship of the optically detectablefeatures of the tracker 108 to the base of the tracker 2302.

Calculating a Tracker Base Geometry of the tracker 106 may also be doneby attaching each of the tracker 106 and Golden Tracker 2106 to atracker mounting location 304 respectively on a characterization jig2202, 2204. The spatial relationship between the two tracker mountinglocations 304 (also referred to as the Inter Tracker Mount SpatialRelationship) must be known for this variation of the embodiment. Usingimage data of both the tracker 106 and the Golden Tracker 2106, theGolden Tracker Definition that further comprises a Golden Tracker BaseGeometry, and the Inter Tracker Mount Spatial Relationship 802,characterization operations can calculate the Tracker Definition 806 andthe Tracker Base Geometry for the tracker (e.g. by performingoptimization operations).

The Inter Tracker Mount Spatial Relationship 802 may be stored in memory206 as a numerical representation of the relative poses between themounting locations represented as rotation matrices, quaternions, eulerangles, translational vectors, Cartesian distances etc. Thecharacterization jig may be manufactured to meet a specific InterTracker Mount Spatial Relationship 802, and the same spatialrelationship may be accessible to the computing unit 104. For example,the numerical representation of this spatial relationship may be datastored in memory 206, preferably in a manner that prevents or reducesthe possibility of deletion. There may be an identifier such as abarcode, QR-code, URL, etc. on the characterization jig itself thatidentifies the Inter Tracker Mount Spatial Relationship 802. Theidentifier may be read using the optical camera 102 and used to look upthe Inter Tracker Mount Spatial Relationship 802 that is stored inmemory 206 or remotely on a network.

Reference is now made to FIGS. 24A and 24B that depict a user method2400A and a corresponding computer-implemented method 2400B to calculatea Tracker Definition 806 using the systems described above. A workflowexecuting on the computing unit 104 instructs the user to place thetracker 106 and Golden Tracker 2106 in a fixed positional relationship2402 to each other by attaching to each other, placing on flat surface2102 or attaching to a characterization jig. The user may then initiatecapture of image data by the optical sensor 102 from multiple vantagepoints 2404. The computing unit 104 receives image data of the tracker106 and the Golden Tracker 2106 from the optical sensor 102, 2404. Thecomputing unit 104 executes instructions to optimize and calculate theTracker Definition 806 for the tracker using the Golden TrackerDefinition.

FIG. 25A depicts the method 2500A as a block diagram of the inputs intocharacterization operations 2504 to calculate a Tracker Definition 806.Using image data 2506 from the camera 102 and the Golden TrackerDefinition 2502, characterization operations 2504 can optimize andcalculate the Tracker Definition 806. The Tracker Definition 806 mayfurther be used as an input to localization operations 2508 to calculatethe pose of the tracker 210 as in the method 2500B illustrated in FIG.25B or be displayed to a user on a display unit 1206.

An exemplary method 2600 of characterization operation using the GoldenTracker 2106 is shown in FIG. 26. The figure describes multiple stagesof the characterization operation 2504. First, image data 2506(comprising multiple 2D images of the tracker and Golden Trackerappearing together) is received from the camera 102 and processed 2606to discern the optically detectable features 108 (e.g., markers) in eachimage. The locations of each optically detectable feature, also calledthe marker locations 2608, in an image are used in the next stage wherethe operations create a correspondence between an optically detectablefeature of one image and the same feature in a second image, and so on2620. In addition to matching markers between each image, each marker isalso classified to determine whether it is part of the Golden Tracker2106 or the tracker 106, 2610. For example, this classification may beperformed by optimizing subsets of matched markers for a best fit withthe Golden Tracker Definition 2502. If the Golden Tracker Definition2502 fits two subsets of matched markers, the characterization operationmay conclude that the Tracker Definition 806 is identical to the GoldenTracker Definition 2502, thus reflecting a scenario where the tracker106 and Golden Tracker 2106 have an identical physical geometry. If not,the operation further uses the matched marker locations of the GoldenTracker 2612 to calculate the pose of the Golden Tracker 2106 betweenthe various images 2616 (thus computing a difference in the variousvantage points of the optical sensor 102). The operation further solves2614 for a Tracker Definition 806 using the matched marker locations ofthe tracker 2612 and the difference in the vantage point from which eachimage was captured (i.e. the relative pose of the Golden Tracker betweeneach image) 2618. Persons skilled in the art will understand that thisis an exemplary operation, and that there may be a variety of operationsthat can be executed to perform such calculations.

Similar to characterization, FIG. 27 illustrates a workflow 2700 where atracker is assessed for accuracy with respect to a Static TrackerDefinition 204. In addition to the Golden Tracker Definition 2502, themethod further utilizes the Static Tracker Definition 204 to optimizethe received image data 2506 and assess the accuracy of the tracker. Itis possible that for some surgical applications, deviation of a tracker106 from a Static Tracker Definition 204 is unacceptable. Throughverification operations 2702, the method may generate a TrackerAssessment Parameter 1006 to quantify this deviation. This parameter maythen be used by the computing unit 104 during a localization procedureto determine whether the tracker 106 being used is accurate or not. Someof the examples of how the Tracker Assessment Parameter 1006 ispresented could be a Boolean flag, a single numerical value or multiplenumerical values associated with how closely the Tracker Definition 806matches the Static Tracker Definition 204, etc.

FIGS. 28A and 28B illustrate a user method 2800A and a correspondingcomputer implemented method 2800B to assess the tracker. A workflowexecuting on the computing unit 104 instructs the user to place thetracker 106 and the Golden Tracker 2106 in a fixed positionalrelationship 2802. The user may then initiate capture of image data 2506by the optical sensor 102 from multiple vantage points 2804. Thecomputing unit 104 receives image data 2506 of the tracker 106 and theGolden Tracker 2106 from the optical sensor 102, 2804. The computingunit 104 executes instructions to optimize and verify the accuracy ofthe tracker 106 with respect to the Static Tracker Definition 204 usingthe Golden Tracker Definition 2502, 2808.

If the tracker 106 and the Golden Tracker 2106 are attached to eachother and the Static Tracker Definition 204 includes the Static TrackerBase Geometry (i.e. the spatial relationship between the opticallydetectable features 108 and the base of the Static Tracker), and theGolden Tracker Base Geometry is known, only one image comprising theGolden Tracker 2106 and tracker 106 is required to assess the accuracyof the tracker 106. Reference is now made to FIG. 29 is a block diagramthat depicts an exemplary method 2900 of performing the verificationoperations 2702. Image data 2506 is processed 2906 to identify thelocation of the markers 2908 in the image data 2506. Using the StaticTracker Definition 204, the Golden Tracker Definition 2502 and themarker locations 2908, the marker locations are then classified 2910 todetermine the location of the tracker markers and the golden trackermarkers, respectively 2912. The actual pose is calculated 2914 using thelocation of the tracker markers and the golden tracker markers 2912, thestatic tracker definition 204 and the golden tracker definition 2502.The expected pose is calculated 2916 using the static tracker definition204, the golden tracker definition 2502, the known static tracker basegeometry 2918 and the golden tracker base geometry 2920 A TrackerAssessment Parameter 1006 is generated by comparing an expected relativepose between the Golden Tracker 2106 and the tracker 106, and an actualrelative pose 2922.

Expanding further on the previously described example of use in theoperating room, the sterile nurse assisting the surgeon during theoperation can handle the tracker 106 and the characterization jig 2202,2204. The second, non-sterile nurse can handle the non-sterile GoldenTracker 2106. The non-sterile nurse can attach the Golden Tracker 2106to one of the two tracker mounting locations 304 on the characterizationjig 2202, 2204. This characterization jig 2202, 2204 may be configuredto allow both the tracker 106 and Golden Tracker 2106 to be attached tothe characterization jig 2202, 2204 simultaneously without compromisingthe sterility of the tracker 106 (i.e. there is no risk of contaminatingthe tracker 106) while allowing the tracker 106 and the Golden Tracker2106 to appear in the field of the view of the optical sensor 102. Forexample, the characterization jig 2202, 2204 may have a tracker mountinglocation 304 on each end of its length for both the tracker 106 and theGolden Tracker 2106 such that the mounting locations 304 are separatedby a sufficient distance to avoid contact between the tracker 106 andGolden tracker 2106 (the distance not being so large that either thetracker 106 or the Golden Tracker 2106 falls outside of the field ofview of the optical sensor 102). For example, as shown in FIGS. 30A and30B the characterization jig 3000 may be a flat plate intended to beused vertically such that there is a tracker mounting location 304 oneach side (e.g. opposite faces of the jig), etc. It may not be necessaryto obtain image data 2506 with both the tracker 106 and the GoldenTracker 2106 in the field of view simultaneously.

It is contemplated that the optical sensor 102 may capture image data2506 serially by capturing a primary image of the tracker 106 attachedto a mounting location 304 and a secondary image of the Golden Tracker2106 attached to another mounting location 304 while the optical sensor102 is positioned at a given vantage point. Such use avoids anypossibility of contamination while the tracker 106 and Golden Tracker2106 are attached or removed from the mounting locations 304 on thecharacterization jig 3000 and the method of use may be repeated tocapture image data 2506 from multiple vantage points. The tracker 106may then be characterized or verified using the image data 2506 and theGolden Tracker Definition 2502 by instructions executing on thecomputing unit 104.

These methods of use where the Golden Tracker 2106 does not have to besterilized may be advantageous. Sterilization may reduce the accuracy ofthe Golden Tracker 2106. This method of use further allowsnon-sterilisable materials to be used to manufacture the Golden Tracker150. In addition, some of the design limitations of a tracker 106 meantfor use in localization do not apply. For example, the Golden Tracker2106 may be made of cheaper materials in order to reduce costs.Alternatively, it may be manufactured using more expensive opticallydetectable features 108 if that lends higher accuracy to themeasurements derived from it.

Tracker Characterization and Verification with Pivotal CharacterizationJig

Systems, methods and components for tracker characterization andverification using a pivotal characterization jig are predominantlydescribed with reference to FIGS. 1-2, 6, and 31-39C though it will beunderstood that there is some overlap with concepts, etc. to earlier orlater described embodiments herein. There is described a systemcomprising: a characterization jig comprising a base, at least twotracker mounting locations and a pivot leg, the pivot leg having a firstend that attaches the pivot leg to the characterization jig and a secondend having a pivot point; and a computer-readable storage device storinginstructions which, when executed on a computing unit, configure thecomputing unit to: receive a first set of articulation images of atracker from an optical sensor attached to a sensor mounting locationwhen the tracker is attached to a first tracker mounting location on thecharacterization jig and the characterization jig is pivoted about thepivot point while the pivot point is mated with a first mating member ofa surface; receive a second set of articulation images of the trackerfrom the optical sensor attached to the sensor mounting location whenthe tracker is attached to a second tracker mounting location on thecharacterization jig and the characterization jig is pivoted about thepivot point while the pivot point is mated with a second mating memberof the surface; and calculate at least one of: A) a Tracker Definitionthat comprises a Tracker Base Geometry using knowledge that the firstset of articulation images and the second set of articulation images areof the tracker being articulated about the pivot point, a spatialrelationship between the first tracker mounting location and the secondtracker mounting location, a Pivot Tracker Mount Spatial Relationship,the first set of articulation images and the second set of articulationimages; and B) a Tracker Assessment Parameter using knowledge that thefirst set of articulation images and the second set of articulationimages are of the tracker being articulated about the pivot point, aspatial relationship between the first tracker mounting location and thesecond tracker mounting location, a Pivot Tracker Mount SpatialRelationship, the first set of articulation images, the second set ofarticulation images, and a Static Tracker Definition comprising a StaticTracker Base Geometry.

The first set of articulation images and the second set of articulationimages may each comprise at least four distinct images of the opticallydetectable features of the tracker.

The pivot feature and at least one of the first mating member and thesecond mating member may further comprise a magnetic component to allowa magnetic connection between them.

The first mating member and second mating member may be coincident.

At least one of the first mating member and the second mating member maybe provided on a sterilization tray containing medical instruments foruse in a surgical procedure.

The characterization jig may comprise an identifier to identify thespatial relationship between the first tracker mounting location and thesecond tracker mounting location of the characterization jig. Theidentifier may be readable by the optical sensor.

The characterization jig may comprise an identifier to identify thePivot Tracker Mount Spatial Relationship. The identifier may be readableby the optical sensor.

The characterization jig may be made entirely of sterilisable material.

The instructions may configure the computing unit to calculate theTracker Assessment Parameter and provide it to a display unit.

The instructions may configure the computing unit to calculate theTracker Assessment Parameter and the system may further comprise adisplay unit to display the Tracker Assessment Parameter.

User instructions may be displayed to a user on a display unit.

The instructions may configure the computing unit to calculate theTracker Definition and the computer readable storage device may furthercomprise instructions to measure a pose of an object attached to thetracker using the Tracker Definition. The object may be a surgical tool.

There is described a system comprising: a characterization jigcomprising a base and at least two tracker mounting locations; anoptical sensor enclosed in a shroud and placed within a clamp, theshroud and the clamp having respective mating surfaces; and acomputer-readable storage device storing instructions which, whenexecuted on a computing unit, configure the computing unit to: receive afirst set of articulation images of a tracker from the optical sensorwhen the tracker is attached to a first tracker mounting location on thecharacterization jig and the optical sensor and shroud is articulatedwithin the clamp; receive a second set of articulation images of thetracker from the optical sensor when the tracker is attached to a secondtracker mounting location on the characterization jig and the opticalsensor and shroud is articulated within the clamp; and calculate atleast one of: A) a Tracker Definition that comprises a Tracker BaseGeometry using knowledge that the first set articulation images and thesecond set of articulation images are obtained from the optical sensorbeing articulated about a center of the clamp, a spatial relationshipbetween the first tracker mounting location and the second trackermounting location, the first set of articulation images and the secondset of articulation images; and B) a Tracker Assessment Parameter usingknowledge that the first set articulation images and the second set ofarticulation images are obtained from the optical sensor beingarticulated about the center of the clamp, the spatial relationshipbetween the first tracker mounting location and the second trackermounting location, the first set of articulation images, the second setof articulation images, and a Static Tracker Definition comprising aStatic Tracker Base Geometry. The first set of articulation images andthe second set of articulation images may each comprise at least fourdistinct images of the optically detectable features of the tracker.

There is provided a computer implemented method comprising the steps of:receiving, by a computing unit, a first set of articulation images of atracker from an optical sensor attached to a sensor mounting locationwhen the tracker is attached to a first tracker mounting location on acharacterization jig comprising a base, at least two tracker mountinglocations and a pivot leg, the pivot leg having a first end thatattaches the pivot leg to the characterization jig and a second end thathas a pivot point, and the characterization jig is pivoted about thepivot point while the pivot point is mated with a first mating member ofa surface; receiving, by the computing unit, a second set ofarticulation images of the tracker from the optical sensor when thetracker is attached to a second tracker mounting location on thecharacterization jig and the characterization jig is pivoted about thepivot point while the pivot point is mated with a second mating memberof the surface; and calculating, by the computing unit, at least one of:A) a Tracker Definition that comprises a Tracker Base Geometry usingknowledge that the first set articulation images and the second set ofarticulation images are of a tracker being articulated about the pivotpoint, a spatial relationship between the first mounting location andthe second mounting location, a Pivot Tracker Mount SpatialRelationship, the first set of articulation images and the second set ofarticulation images; and B) a Tracker Assessment Parameter usingknowledge that the first set articulation images and the second set ofarticulation images are of the tracker being articulated about the pivotpoint, a spatial relationship between the first mounting location andthe second mounting location, a Pivot Tracker Mount SpatialRelationship, the first set of articulation images, the second set ofarticulation images, and a Static Tracker Definition. The first set ofarticulation images and the second set of articulation images may eachcomprise at least four distinct images of the optically detectablefeatures of the tracker.

The method may further comprise providing, by the computing unit, atleast one of the Tracker Definition or the Tracker Assessment Parameteras calculated for use in a localization procedure.

There is described a characterization jig comprising a base, the basehaving a top side and a bottom side, the top side comprising at leasttwo tracker mounting locations and the bottom side comprising a pivotleg, the pivot leg further comprising a first end to attach to thebottom side of the characterization jig and a second end comprising apivot point about which the characterization jig can be articulated.

In the characterization jig, the first end of the pivot leg may be ahinge joint movable in one degree of freedom and the bottom side has aslot customized to accommodate a length of the pivot leg.

This document describes two systems—one for localization and the otherfor characterization. It is to be understood that these may be distinctsystems that utilise the same hardware such as the optical sensor tocapture images and the computing unit to execute instructions. Thesesystems may also be distinct systems that use different hardware. It isalso possible that these distinct systems communicate with each other topresent a seamless user experience but optionally utilise the samehardware. Alternatively, the characterization system may be usedseparately from the localization system. For example, thecharacterization system may be used as part of quality control at amanufacturing site while manufacturing a localization system for use ina surgical setting. The hardware and software used to characterize thetracker may be completely distinct.

A characterization and/or verification system described below usesimages of a tracker comprising optically detectable features that wereobtained from the optical sensor while either the optical sensor ortracker is articulated about a geometrical constraint that allows acomputing unit to execute operations to calculate the value of theTracker Definition or Tracker Assessment Parameter.

In one such example of a characterization system, a characterization jigmay be used with the components of the localization system such as theoptical sensor (e.g., a camera) and the tracker with opticallydetectable features (e.g., markers). The camera may be the same hardwareused for localization or may be a part of a computing unit or laptope.g., a webcam on a laptop. The camera may be attached to a platform.

The characterization jig comprises a rigid base with a top side and abottom side, a pivot leg with two ends, and at least two mountinglocations on the top side upon which a tracker can be attached. Twoobjects are “attached” when both are in contact with each other to forma connection and there is a holding mechanism to enforce the connection.The attachment may be rigid and removable, for example, selectivelyremovable. It may also be repeatable by virtue of the type of mechanismused for attachment, for e.g., a kinematic mount.

The pivot leg is a shaft-like attachment (temporary or permanent) to thejig and has two ends. A first end attaches the pivot leg to the bottomside of the jig, opposite the top side of the jig that comprises trackermounting locations. A second end comprises a pivot point (e.g. a tip)such that the entire characterization jig can be articulated about thepoint. In this document, “articulate” is a verb referring to themovement of an object (or a part of it) within a constraint such thatthe movement is restricted about a single center of rotation. Multiplemethods of attachment of the pivot leg to the jig are contemplated. Forexample, the pivot leg may be a standalone piece that is attached to thejig using a helical thread running around the first end; the pivot legmay be attached to the jig as a hinge joint movable in one degree offreedom and foldable into a slot customized to accommodate the length ofthe pivot leg, the slot being located on the bottom side of thecharacterization jig; there may be a strong magnetic connection betweenthe first end of the pivot leg and the bottom side of thecharacterization jig to allow a user to articulate the jig about thepivot point without slippage, etc.

The jig may be made of sterilisable material to allow use within anoperating room or any such sterilized environment. The geometry of thecharacterization jig is known to the computing unit. This implies thatthe physical shape, size, features etc. of the characterization jig areknown since the jig may be manufactured under strict tolerances. Thecharacterization and/or verification operations described herein utilizeknown spatial relationships between the various components of thecharacterization jig.

These known spatial relationships or any of the other geometricalcharacteristics of the jig may be pre-loaded into the memory of thecomputing unit of the characterization system (e.g. as a hard-codedvalue, as a 3D model, etc.) and used in the calculation of the TrackerDefinition or the calculation of a Tracker Assessment Parameter toverify of the accuracy of a tracker with respect to a Static TrackerDefinition. The Tracker Assessment Parameter may be displayed to a useron a display unit. Using this configuration of the jig, the computingunit may optionally calculate a Tracker Base Geometry (i.e. how theoptically detectable features relate to the base of the tracker) as partof the Tracker Definition. It may also be possible to verify theaccuracy of the tracker with respect to the Static Tracker Definitionthat comprises a Static Tracker Base Geometry.

An Inter Tracker Mount Spatial Relationship (i.e. the spatialrelationship between two tracker mounting locations on the jig) and aPivot Tracker Mount Spatial Relationship (i.e. the spatial relationshipbetween any one of the tracker mounting locations and the pivot point onthe second end of the pivot leg of the jig) may be stored in memory as anumerical representation of the relative poses between the mountinglocations represented as rotation matrices, quaternions, euler angles,translational vectors, Cartesian distances etc. The characterization jigcomprising the pivot leg may be manufactured to meet a specific InterTracker Mount Spatial Relationship and/or Pivot Tracker Mount SpatialRelationship. The same spatial relationships may be accessible to thecomputing unit. For example, the numerical representation of the spatialrelationships may be data stored in memory, preferably in a manner thatprevents or reduces the possibility of deletion. There may be anidentifier such as a barcode, QR-code, URL, etc. on the characterizationjig itself that identifies the relationships. The identifier may be readusing the optical sensor and used to look up the relationships that arestored in memory or remotely on a network.

To characterize the tracker of verify the accuracy of the tracker, thecamera captures images specifically for characterization (also called“articulation image data” in this document) of the tracker when thetracker is attached to the tracker mounting locations on thecharacterization jig, and the jig is articulated about the pivot pointof the pivot leg. Using known spatial relationships of thecharacterization jig (such as the Inter Tracker Mount SpatialRelationship and the Pivot Tracker Mount Spatial Relationship) andknowing that the articulation image data is obtained while the jig ispivoted about the pivot point, the computing unit can executeinstructions to calculate a Tracker Definition or a Tracker AssessmentParameter. When articulated about a point, a person skilled in the artwill understand that a minimum of four distinct images during eacharticulation will be required in order to calculate a center of asphere. The characterization and verification processes are furtherdescribed below.

Reference is now made to FIG. 31 that illustrates the characterizationjig 3100 with the pivot leg 3104 comprising a pivot point 3106 affixedto the rigid base 3102. The pivot leg 3104 is further configured to matewith a mating member on a rigid surface. The mating member on the rigidsurface may be such that a user can access it repeatedly to articulatethe characterization jig 3100.

The characterization jig 3100 is not to be limited to the examples shownabove. For example, a surgical probe with a tip may be articulated aboutthe tip within a hemi-spherical divot on the surface. In addition to itsutility in the process of tracker characterization, the probe may haveadditional utility for navigation or localization. The probe may providea first mounting location 304 for the tracker 106. A second mountinglocation 304 may be provided by a separate attachment to the probe thatattaches in a fixed and repeatable manner to form a characterization jig3100 with a known Inter Tracker Mount Spatial Relationship 802 betweenthe mounting locations and a known Pivot Tracker Mount SpatialRelationship.

Reference is now made to FIG. 32 showing a configuration of a system3200 using the characterization jig 3100 with at least two trackermounting locations 304 and the pivot leg 2104 as an example for clarity.The pivot point 3106 of the pivot leg 2104 is mated with the matingmember 3202 on the rigid surface 3204. A user attaches the opticalsensor 102 (e.g.: a camera) to a sensor mounting location 306 orplatform 2306. The optical sensor 102 is in communication with acomputing unit 104. The computing unit 104 may provide correspondinguser instructions. The sensor mounting location 306 or platform 2306 mayoptionally be on the same rigid surface 205 as the mating member 3202.The spatial relationship between the sensor mounting location 306 orplatform 2306 and the mating member 3202 is not required to be known tothe computing unit 104. The user attaches the tracker 106 to the trackermounting locations 304 on the characterization jig 3100 and articulatesthe characterization jig 3100 while the tracker 106 is within a field ofview of the optical sensor 102. Reference is now made to FIG. 33 depictsthe configuration of a system 3300 where the characterization jig 3100is being articulated while the pivot point 3106 is mated with the matingmember 3202 on the rigid surface 3204. As the characterization jig 3100is articulated within the mating member 3202, the optical sensor 102captures articulation image data.

In further variations of this configuration of the system, the matingmember 3202 and the pivot point 3106 may each comprise a magneticcomponent to allow the mating member 3202 and the pivot point 3106 ofthe characterization jig 3100 to mate magnetically. This allows arepeatable and stable mating connection between the mating member 3202and the characterization jig 3100. The magnetic force may further assista user in keeping the pivot point 3106 engaged with the mating member3202 during articulation, thus potentially reducing errors introduceddue to slippage of the pivot point 3106 within the mating member 3202while articulating. The mating member 3202 may be provided on asterilization tray containing medical instruments for use in a surgicalprocedure. The pivot leg 2104 and the mating member 3202 should beinterpreted as broadly as possible. For example, while FIG. 32 depictsthe pivot leg 2104 of the characterization jig 3100 and the matingmember 3202 as a part of the rigid surface 3204, the locations orgeometrical shapes of these components may be reversed such that thecharacterization jig 3100 comprises the mating member 3202 and the pivotleg 2104 is attached to the rigid surface 3204. This would allow thecharacterization jig 3100 to be stationary while the surface 3204 (forexample, in the form of a rectangular tray with a sensor mountinglocation 306) is articulated about the pivot point 3106.

Reference is now made to FIGS. 34A and 34B. In a method of use and acorresponding computer implemented method to obtain a Tracker Definition806, a camera 102 is rigidly attached to a sensor mounting location 306and connected to a computing unit 104. In a first step 3402: A userattaches a tracker 106 (with an unknown Tracker Definition 806) to afirst mounting location 304 on the characterization jig 3100; The usermates the characterization jig through its pivot leg 203 within a matingmember 3202 on a rigid surface 3204 and articulates the characterizationjig 3100, while the tracker 106 is in a field of view of the camera 102.The computing unit 104 receives a first set of articulation images 3406.In a second step 3404: The user then attaches the tracker 106 to asecond tracker mounting 304 and articulates the characterization jig3100 while the tracker 106 is in the field of view of the camera 102.The computing unit 104 receives a second set of articulation images3408. Knowing that both sets of articulation image data is obtained byarticulating the characterization jig 3100 about a fixed point, andusing both sets of articulation image data, the Inter Tracker MountSpatial Relationship 802, and the Pivot Tracker Mount SpatialRelationship 3612, the computing unit 104 executes instructions tocalculate a Tracker Definition 806 for the tracker 106 usingcharacterization operations 3410. This Tracker Definition 806 may alsocomprise a Tracker Base Geometry.

There may be multiple mating members 3202 available for use such thatthe characterization jig 3100 is placed within a first mating member3202 to obtain a first set of articulation images, and a second matingmember 3202 to obtain a second set of articulation images. It is notnecessary to use the same mating member 3202 since the Pivot TrackerMount Spatial Relationship 3612 is known. The first and second matingmembers 204 may optionally be coincident as shown in the figures of thisdocument.

The sensor mounting location 306 and the tracker mounting locations 304may be reversed such that there are two sensor mounting locations 306 onthe characterization jig 3100 (in which case, the tracker 106 must bestationary while the articulation image data is captured). This may beadvantageous where the tracker 106 does not have a base 2302 (e.g., ifthe tracker 106 is integrally formed with a tool), or where multipletrackers 106 are to be simultaneously characterized. The instructionsexecuting on the computing unit 104 may be modified accordingly suchthat the calculated result includes the spatial relationship between theoptical sensor 102 and its base, in addition to the Tracker Definition806 (excluding Tracker Base Geometry). The calculation of this spatialrelationship of the optical sensor 102 may further be useful inscenarios where the optical sensor 102 is attached to a tool. When thespatial relationship between an effector of a tool and the sensorattachment location is known and/or accessible to the computing unit 104during navigation, the computing unit 104 may perform navigation withrespect to the effector of the tool. FIGS. 35A-35D illustrate a group ofsensor systems (3500A, 3500B, 3500C, 3500D) where an optical sensor 102is attached to various tools, including a probe 1612, a robotmanipulator 3502, a broach handle 3504 and a calibration tool 3506respectively.

The computing unit 104 may comprise multiple, distributed processingunits (e.g. characterization may be carried out on one computing unit104, whereas intra-operative localization may be carried out by anothercomputing unit 104). It will be evident to those skilled in the art whatdata must be transferred between the various computing units 104 toenable the functionality of the systems and methods described herein.Tracker characterization or verification may be performed on an ad-hocbasis as part of routine maintenance of trackers 106, or may beperformed prior to each localization procedure that uses a tracker 106.A person skilled in the art will appreciate that the level of accuracyof measurements will improve with an increase in the number of mountinglocations 306 for the camera 102 and the corresponding articulationimage data captured by the camera 102.

Reference is now made to FIG. 36A which illustrates the method 3600A andthe flow of information to determine the Tracker Definition 806 of thetracker 106. Using the methods described above to obtain articulationimage data 3602, characterization operations 3604 may calculate theTracker Definition 806 using the Inter Tracker Mount SpatialRelationship 3610 and the Pivot Tracker Mount Spatial Relationship 3612,for example, by optimizing for the Tracker Definition 806 using thespatial relationships as constraints. FIG. 35B illustrates the output ofa characterization operation 3604 being used in localization operations3608 to calculate a pose of a tracker 210. As mentioned previously, thecharacterization jig 3100 may be configured as a probe 1612 that hasutility in localization as well.

A exemplary method 3700 of the characterization operation 3604 is shownin FIG. 37 for clarity. First, articulation image data 3602 (comprisingimage sets of articulation images of the tracker 106 when the tracker iscoupled to the characterization jig 3100 and the characterization jig3100 is articulated about the pivot point 3106) is received from thecamera 102 and processed 3706 to identify the location of the opticallydetectable features 3708 (e.g., markers) in each image. Note: Thearticulation of the characterization jig 3100 may be arbitrary withrespect to direction/magnitude, or there may be some guidance providedto a user by the computing unit 104 to ensure that the articulationimage data 3602 has the appropriate characteristics (e.g. number ofimages, angular span, direction, magnitude, etc.) to allow the computingunit 104 to execute instructions to accurately solve for the TrackerDefinition 806. The operation also generates and maintains anintra-image set marker correspondence 3710 such that across each imagein the articulation image set 3602, the optically detectable features108 are associated with the same physical feature. Several methods forgenerating and maintaining the intra-image set marker correspondence3710 may be used, and are evident to those skilled in the art. Thisprocess is repeated for each articulation image set 3602. The markerlocations 3708 of each optically detectable feature 108, as well as theintra-image set marker correspondence 3710 are used in the next stagewhere, in conjunction with the Pivot Tracker Mount Spatial Relationship3612 (i.e. how any one tracker mounting location 304 spatially relatesto the pivot point 3106 of the characterization jig 3100), theInter-Tracker Mount Spatial Relationship 3610 (i.e. how the trackermounting locations 304 spatially relate to each other) and the knowledgethat the articulation image sets 3602 are of a tracker 106 (attached toa characterization jig 3100) being articulated about a pivot point 3106,the Tracker Definition 806 is calculated 3712. The calculation of theTracker Definition 806 (i.e. how the optically detectable features ofthe tracker 106 relate to each other) may include calculation of theTracker Base Geometry (i.e. how the optically detectable features 108 ofthe tracker relate to the base of the tracker 2302). This calculationmay utilize optimization routines where the unknown Tracker Definition806 is being optimized, based on the marker locations 3708 in thearticulation image sets 3602, the intra-image set marker correspondence3710, and further utilizing the Pivot Tracker Mount Spatial Relationship3612, the Inter Tracker Mount Spatial Relationship 3610 and theknowledge of the articulation being about a fixed point as optimizationconstraints. To solve for the Tracker Definition 806, it may benecessary to simultaneously solve for the inter-image set markercorrespondence 3710.

Reference is now made to FIG. 38A where there is illustrated the method3800A and flow of information through verification operations 3802 todetermine the accuracy of a tracker 106 with respect to a Static TrackerDefinition 204. For example, verification operations 3802 may be set upas optimization operations that minimize error between the articulationimage data 3602 and a Static Tracker Definition 204 further comprising aStatic Tracker Base Geometry 2918 (if applicable). The Inter TrackerMount Spatial Relationship 802 and the Pivot Tracker Mount SpatialRelationship 3612 provide optimization constraints. The operations maydetermine whether the Tracker Definition 806 comprising Tracker BaseGeometry of a given tracker 106 is within a prescribed tolerance of theStatic Tracker Definition 204 comprising Static Tracker Base Geometry2918. The operations may further calculate a Tracker AssessmentParameter 1006 to quantify the deviation of the Tracker Definition 806from the Static Tracker Definition 204. FIG. 38B illustrates a systemwhere the output of a verification operation 3802 being used inlocalization operations 3808 to calculate pose of a tracker 210. Thelocalization operations 3808 may use or discard the Static TrackerDefinition 204, depending on the Tracker Assessment Parameter 1006.

In another example of a characterization system, the optical sensor isarticulated instead of the tracker attached to the jig. The opticalsensor 102 is enclosed within a spherical shroud 1604 and clamp 1602 asillustrated in FIGS. 39A-39C. The optical sensor 102 (secured within theshroud 1604) may be articulated by virtue of its shape within theconstraints of a geometry of the shroud 1604 and clamp 1602. Forexample, if the outer surface of the shroud 1604 and the inner surfaceof a clamp 1602 have curved mating surfaces (which curves defineportions of a sphere), the optical sensor 102 will be articulated abouta geometrical point that corresponds to the center of the sphere i.e.the center of the clamp 1602. As seen in system 3900 illustrated in FIG.39C, the movement of the optical sensor 102 and shroud 1604 within theclamp 1602 allows the computing unit 104 to obtain articulation imagedata 3602 of the tracker 106. The tracker 106 may be attached to trackermounting locations 304 on a characterization jig 3902 sequentially toobtain articulation image data 3602. The tracker mounting locations 304may be provided by a characterization jig 160 with a known Inter TrackerMount Spatial Relationship 802. Knowing that the articulation image data3602 is obtained by articulating about a point, the system may be usedto execute characterization operations 3604 to determine the TrackerDefinition 806 of the tracker 106 using the articulation image data3602, the Inter Tracker Mount Spatial Relationship 802 and the spatialrelationship between the point (for e.g., the center of the clamp) andat least one of the tracker mounting locations 304. This spatialrelationship is analogous to the Pivot Tracker Mount SpatialRelationship 3612 of the previous embodiment where the characterizationjig 3100 comprises the pivot leg 2104. The calculated Tracker Definition806 comprises the Tracker Base Geometry of the tracker 106 whichdescribes how the optically detectable features 108 of the tracker 106spatially relate to the base of the tracker 2302. As discussed above,this is useful in scenarios where there is a desire to know how thetracker 106 spatially relates to an object that it is attached to,through its base 2302 such that the localization system can providerelevant spatial measurements of the object.

Tracker Characterization with Inertial Sensor

Systems, methods and components for tracker characterization using aninertial sensor are predominantly described with reference to FIGS. 1-2,6, and 40A-46 though it will be understood that there is some overlapwith concepts, etc., to earlier or later described embodiments herein.There is provided a system comprising: a computer-readable storagedevice storing instructions and/or data which, when executed on acomputing unit, configures the computing unit to: receive a first imageof a tracker from an optical sensor at a first vantage point and a firstinertial measurement from one or more inertial sensors, wherein the oneor more inertial sensors are co-registered to the optical sensor andwherein the first image and first inertial measurement are obtained bythe respective optical sensor and one or more inertial sensors when thetracker is at a resting position in a field of view of the opticalsensor; receive a second image of the tracker from the optical sensor ata second vantage point and a second inertial measurement from the one ormore inertial sensors, wherein the second image and second inertialmeasurement are obtained by the respective optical sensor and one ormore inertial sensors when the tracker is at the resting position in afield of view of the optical sensor; and calculate at least one of: aTracker Definition using the first image, the second image and adifference in orientation in at least 2 DOF between the first vantagepoint and the second vantage point; and a Tracker Assessment Parameterusing the first image, the second image, the difference in orientationin at least 2 DOF between the first vantage point and the second vantagepoint, and a Static Tracker Definition.

The computing unit may be further configured to calculate the differencein orientation in at least 2 DOF between the first vantage point and thesecond vantage point using the first inertial measurement and the secondinertial measurement.

Each of the one or more inertial sensors comprises any of anaccelerometer, a gyroscope and a magnetometer.

The computing unit may be configured to: calculate a Tracker Definition;calculate a Tracker Base Geometry using the first image, the firstinertial measurement, the second image, the second inertial measurementand the resting position of the tracker; and provide the Tracker BaseGeometry for use in a localization procedure.

The computing unit may be configured to provide one of the TrackerDefinition and the Tracker Assessment Parameter for use in alocalization procedure.

The computing unit may be further configured to provide userinstructions to move the sensor apparatus from the first vantage pointto the second vantage point for display on a display unit.

The computing unit may be configured to calculate the Tracker AssessmentParameter and is further configured to provide the Tracker AssessmentParameter for display on a display unit. The system may further comprisethe display unit to display the Tracker Assessment Parameter.

The system may further comprise the tracker.

The system may further comprise the optical sensor.

The system may further comprise the one or more inertial sensors.

There is provided a computer-implemented method comprising: receiving,by a computing unit, a first image of a tracker from an optical sensorat a first vantage point and a first inertial measurement from one ormore inertial sensors, wherein the one or more inertial sensors areco-registered to the optical sensor and wherein the first image andfirst inertial measurement are obtained by the respective optical sensorand one or more inertial sensors when the tracker is at a restingposition in a field of view of the optical sensor; receiving, but thecomputing unit, a second image of the tracker from the optical sensor ata second vantage point and a second inertial measurement from the one ormore inertial sensors, wherein the second image and second inertialmeasurement are obtained by the respective optical sensor and one ormore inertial sensors when the tracker is at the resting position in afield of view of the optical sensor; and calculating, by the computingunit, at least one of: a Tracker Definition using the first image, thesecond image and a difference in orientation in at least 2 DOF betweenthe first vantage point and the second vantage point; and a TrackerAssessment Parameter using the first image, the second image, thedifference in orientation in at least 2 DOF between the first vantagepoint and the second vantage point and a Static Tracker Definition.

The method may further comprise calculating the difference inorientation in at least 2 DOF between the first vantage point and thesecond vantage point using the first inertial measurement and the secondinertial measurement.

There is provided a system comprising: a computer-readable storagedevice storing instructions and/or data which, when executed on acomputing unit, configures the computing unit to: receive a first imageof a tracker from an optical sensor at a first vantage point and a firstinertial measurement from one or more inertial sensors, wherein the oneor more inertial sensors are co-registered to the optical sensor andwherein the first image and first inertial measurement are obtained bythe respective optical sensor and one or more inertial sensors when thetracker is at a resting position in a field of view of the opticalsensor; and calculate a Tracker Assessment Parameter using the firstimage, the first inertial measurement, the resting position of thetracker, and a Static Tracker Definition.

The data stored by the computer readable storage device may comprise theStatic Tracker Definition. The Static Tracker Definition may comprise aStatic Tracker Base Geometry.

The computing unit may be further configured to provide the TrackerAssessment Parameter for use in a localization procedure.

The computing unit may be further configured to provide userinstructions to capture the first image for display on a display unit.

The computing unit may be further configured to provide the TrackerAssessment Parameter for display on a display unit.

The system may further comprise a display unit to display the TrackerAssessment Parameter.

There is provided a computer-implemented method comprising: receiving,by a computing unit, a first image of a tracker from an optical sensorat a first vantage point and a first inertial measurement from one ormore inertial sensors, wherein the one or more inertial sensors areco-registered to the optical sensor and wherein the first image andfirst inertial measurement are obtained by the respective optical sensorand one or more inertial sensors when the tracker is at a restingposition in a field of view of the optical sensor; and calculating, bythe computing unit, a Tracker Assessment Parameter using the firstimage, the first inertial measurement, the resting position of thetracker, and a Static Tracker Definition.

This document describes two systems—one for localization and the otherfor characterization. It is to be understood that these may be distinctsystems that utilise the same hardware such as the camera to captureimages and the computing unit to execute instructions. These systems mayalso be distinct units that use different hardware. It is also possiblethat these distinct systems communicate with each other to present aseamless user experience but optionally utilise the same hardware.Alternatively, the characterization system may be used separately fromthe localization system. For example, the characterization system may beused as part of quality control at a manufacturing site whilemanufacturing a localization system for use in a surgical setting. Thehardware and software used to characterize the tracker may be completelydistinct. Tracker characterization may be performed on an ad-hoc basisas part of routine maintenance of trackers, or may be performed prior toeach localization procedure that uses a tracker.

In an embodiment to calculate a Tracker Definition, the sensor apparatusfurther comprises one or more an inertial sensors, in addition to theoptical sensor. The inertial and optical sensors are co-registered toeach other, i.e. the coordinate frame of the optical sensor (in whichoptical measurements are calculated) is known to the computing unit withrespect to the coordinate frame of the inertial sensor (in whichinertial measurements are calculated). This can be done through asuitable calibration method, several of which are known in the art. Eachof the one or more inertial sensors may be an accelerometer, gyroscope,magnetometer, etc. As an example, this document describes the use of anaccelerometer as the inertial sensor to provide inertial measurementswith respect to gravity.

Reference is now made to FIGS. 40A and 40B illustrating twoconfigurations of a system 4000A, 4000B using the sensor apparatus 4002comprising a camera (optical sensor) 102 and an accelerometer (inertialsensor) 4004. The sensor apparatus 138 is connected to a computing unit104. The sensor apparatus 138 may be handheld as it is moved to variousvantage points around a tracker 106 such that the view of the tracker106 is substantially different from each vantage point. A person skilledin the art will appreciate that the sensor apparatus 138 is movedsteadily to each vantage point in order to avoid any unexpected andrandom movement introduced by a user that may affect the accuracy ofmeasurements. From each vantage point, the computing unit 104 isprovided image data and inertial measurements required forcharacterization. The tracker 106 is placed in a resting position suchthat an unoccluded view of the optically detectable features 108 of thetracker 106 is seen from the optical sensor 102. The tracker 106 mustremain in the same resting position while multiple optical and inertialmeasurements are captured. A minimum of two images and two correspondinginertial measurements 4206 from two different vantage points arerequired for this configuration to calculate a Tracker Definition 806.However, a person skilled in the art will understand that multipleoptical and inertial measurements may be used to improve accuracy. Theinertial measurements 4206 are used as inputs into characterizationoperations 4204 to determine a difference in orientation in at least 2DOF between the vantage points 4618, along with corresponding opticalmeasurement at the same timestamp, in order to calculate a TrackerDefinition 806 for the tracker 106.

If a Tracker Base Geometry is to be calculated, the tracker 106 may beplaced in a known resting position with respect to the inertialmeasurement of gravity e.g.: tracker 106 may be placed upright on arigid surface 4006 through its base as illustrated in FIG. 40A. Usingthe additional knowledge of the resting position of the tracker withrespect to the inertial measurements 4206, the computing unit 104 maycalculate the Tracker Base Geometry in addition to the TrackerDefinition 806. In an alternative configuration illustrated in FIG. 40B,the tracker 106 is placed on a flat surface 4006 such that its opticallydetectable features are facing up.

FIGS. 41A and 41B illustrate a user method 4100A and a correspondingcomputer implemented method 4100B to calculate a Tracker Definition 806.A workflow executing on the computing unit 104 instructs the user tocapture a first image of a tracker 106 from an optical sensor 102 and afirst inertial measurement from an inertial sensor 4004 co-registeredwith the optical sensor 102 from a first vantage point 4102. The user isthen instructed to capture a second image of a tracker 106 from anoptical sensor 102 and a second inertial measurement from an inertialsensor 4004 co-registered with the optical sensor 102 from a secondvantage point 4104. The computing unit 104 executes instructions toreceive the optical and inertial measurements from a co-registeredoptical sensor and inertial sensor 4106, 4108. Using the opticalmeasurements in combination with a difference in orientation in at least2 DOF between each vantage point derived from the inertial measurements,the computing unit 104 is able to calculate a Tracker Definition 806 forthe tracker 106, 4110. This calculated Tracker Definition 806 comprisesthe spatial relationship between the optically detectable features 108of the tracker.

FIG. 42 illustrates a system 4200 using a block diagram to show theinputs into a characterization operation 4204 to calculate a TrackerDefinition 806 comprising optical measurements 4202 and inertialmeasurement 4206 captured by a sensor apparatus 138 comprising anoptical sensor 102 and an inertial sensor 4004. FIG. 43 furtherillustrates a system 4300 using a block diagram depicting the TrackerDefinition 806 (and optionally, the Tracker Base Geometry) being used inlocalization operations 4308 to calculate a pose of the tracker 210.This diagram also illustrates the optical sensor 102 providing opticalmeasurements 4202 for characterization operation 4204 and a real timefeed of camera images 202 for localization operations 4308.

It is possible that for some surgical applications, deviation (e.g. bymore than a threshold amount) of a tracker 106 from a Static TrackerDefinition 204 is unacceptable. The system may generate a TrackerAssessment Parameter 1006 to quantify this deviation. This TrackerAssessment Parameter 1006 may then be used by the computing unit 104during a localization procedure to determine whether the tracker 106being used is accurate or not. Some of the examples of how the TrackerAssessment Parameter 1006 is presented could be a Boolean flag, a singlenumerical value or multiple numerical values associated with how closelythe Tracker Definition 806 matches the Static Tracker Definition 204,etc. FIG. 44 illustrates a system 4400 as a block diagram to show theinputs into a verification operation 4402 to calculate the TrackerAssessment Parameter 1006 that is used to verify the accuracy of atracker with respect to a Static Tracker Definition 204. The TrackerAssessment Parameter 1006 may simply be presented to the user (e.g.: ona display unit 1206) to allow the user to make further decisions. Theaccuracy of a tracker can be verified by comparing its calculatedTracker Definition 806 to a Static Tracker Definition 204. A tracker mayalso be verified by comparing actual optical measurements 4202 (incombination with a change in orientation in at least 2 DOF between eachvantage point calculated using the inertial measurements 4206) withexpected optical measurements calculated using a Static TrackerDefinition 204, and further based on the change in orientation in atleast 2 DOF between each vantage point 4618.

In the exemplary configuration illustrated in FIG. 40A, the computingunit 104 may only require a single inertial measurement 4206, thecorresponding optical measurement 4202, and the known resting positionof the tracker 106 with respect to the inertial measurement 4206 toverify the accuracy of the tracker 106 with respect to the StaticTracker Definition 204. The computing unit 104 may further verify theaccuracy of the tracker 106 with respect to a Static Tracker BaseGeometry 2918, if provided. FIG. 45 illustrates a system as a blockdiagram depicting the Tracker Assessment Parameter 1006 being used inlocalization operations 7308 to calculate a pose of the tracker 210.

FIG. 46 depicts an exemplary method 4600 of the characterizationoperation 4204. The figure describes three stages of thecharacterization operation 4204. First, optical measurements 4202(comprising multiple 2D images) are received from the optical sensor 102and processed 4606 to identify the optically detectable features (e.g.,markers) 108 of the tracker 106 in each image. The locations of eachoptically detectable feature, also called the marker locations 4608, inall the images 4202 are used in the next stage where the operationscreate a correspondence between an optically detectable feature of afirst image and the same feature in a second image, and so on 4610.Meanwhile, the inertial measurements 4206 from the inertial sensor 4004are used by the orientation calculator 4616 to calculate a difference inthe orientation in at least 2 DOF between each vantage point from wherea corresponding image was captured 4618. Using this difference and thematched marker locations 4612, the operation solves 4614 for a value fora Tracker Definition 806. Persons skilled in the art will understandthat this is an exemplary operation, and that there may be a variety ofoperations that can be executed to perform such calculations.

Accordingly, it is to be understood that this subject matter is notlimited to particular embodiments described, and as such may vary. It isalso to be understood that the terminology used herein is for thepurpose of describing particular embodiments only, and is not intendedto be limiting.

As will be apparent to those of skill in the art upon reading thisdisclosure, each of the individual embodiments described and illustratedherein has discrete components and features which may be readilyseparated from or combined with the features of any of the other severalembodiments without departing from the teachings herein. Any recitedmethod can be carried out in the order of events recited or in any otherorder which is logically possible.

Various embodiments have been described herein with reference to theaccompanying drawings. It will, however, be evident that variousmodifications and changes may be made thereto, and additionalembodiments may be implemented, without departing from the broader scopeof the disclosed embodiments as set forth in the claims that follow.

1. A system comprising: a Golden Tracker; and a computer-readablestorage device storing instructions and data, including a Golden TrackerDefinition for the Golden Tracker, which, when executed on a computingunit, configure the computing unit to: receive a first image of atracker and the Golden Tracker, separated by a fixed spatialrelationship, the first image captured by an optical sensor from a firstvantage point when both trackers appear in a field of view of theoptical sensor; receive a second image of the tracker and the GoldenTracker, separated by the fixed spatial relationship, the second imagecaptured by the optical sensor from a second vantage point when bothtrackers appear in the field of view of the optical sensor; andcalculate at least one of: a Tracker Definition for the tracker usingthe first image, the second image, an image difference, and the GoldenTracker Definition; and a Tracker Assessment Parameter for the trackerusing the first image, the second image, the image difference, and aStatic Tracker Definition.
 2. The system of claim 1 wherein thecomputing unit is further configured to provide the at least one of theTracker Definition or the Tracker Assessment Parameter which iscalculated for use in a localization procedure.
 3. The system of claim 1wherein the computing unit is further configured to provide the TrackerAssessment Parameter when calculated for display on a display unit. 4.The system of claim 3 wherein the system further comprises a displayunit to display the Tracker Assessment Parameter.
 5. The system of claim1 wherein the computing unit is further configured to calculate theimage difference by calculating a difference in pose of the GoldenTracker between the first image and the second image using the firstimage, the second image and the Golden Tracker Definition.
 6. The systemof claim 1 wherein the computing unit is further configured to provideuser instructions for display on a display unit.
 7. The system of claim1 wherein the tracker and the Golden Tracker are attached to arespective mounting location on a characterization jig.
 8. The system ofclaim 7 wherein the characterization jig is made entirely ofsterilisable material.
 9. The system of claim 7 wherein thecharacterization jig is made entirely of non-sterilisable material. 10.The system of claim 7 further comprising the characterization jig. 11.The system of claim 1 further comprising the optical sensor.
 12. Thesystem of claim 1 wherein the fixed spatial relationship is an InterTracker Mount Spatial Relationship of a characterization jig comprisingmounting locations for the tracker and Golden Tracker, wherein, when thecomputing unit is configured to calculate the Tracker Definition, thecomputing unit is further configured to calculate a Tracker BaseGeometry using the first image, the second image, the image difference,the Inter Tracker Mount Spatial Relationship, the Golden TrackerDefinition and a Golden Tracker Base Geometry.
 13. The system of claim 1wherein the optical sensor is attached to a platform and the tracker andthe Golden Tracker are attached to each other.
 14. The system of claim 1wherein, when the computing unit is configured to calculate the TrackerDefinition, the computing unit is further configured to calculate aTracker Base Geometry using the first image, the second image, the imagedifference, the fixed spatial relationship, the Golden TrackerDefinition and a Golden Tracker Base Geometry.
 15. The system of claim14 wherein the tracker and the Golden Tracker are attached to eachother.
 16. A system for characterizing a tracker for localization in asurgical procedure, the system comprising: a non-sterile Golden Tracker;a sterile characterization jig comprising at least two mountinglocations and further configured to simultaneously attach the GoldenTracker and a sterile tracker on each mounting location respectivelywithout compromising the sterility of the tracker; a computer-readablestorage device storing instructions and data, including a Golden TrackerDefinition for the Golden Tracker, which, when executed on a computingunit, configure the computing unit to: receive a first image of thetracker and the Golden Tracker captured by an optical sensor from afirst vantage point when both trackers appear in a field of view of theoptical sensor; receive a second image of the tracker and the GoldenTracker captured by the optical sensor from a second vantage point whenboth trackers appear in the field of view of the optical sensor; andcalculate at least one of: a Tracker Definition for the tracker usingthe first image, the second image, an image difference, and the GoldenTracker Definition; and a Tracker Assessment Parameter for the trackerusing the first image, the second image, the image difference, theGolden Tracker Definition and a Static Tracker Definition.
 17. Acomputer-implemented method comprising the steps of: receiving, by acomputing unit, a first image of a tracker and a Golden Tracker,separated by a fixed spatial relationship, the first image captured byan optical sensor from a first vantage point when both trackers appearin a field of view of the optical sensor; receiving, by the computingunit, a second image of a tracker and a Golden Tracker, separated by thefixed spatial relationship, the second image captured by the opticalsensor from a second vantage point when both trackers appear in thefield of view of the optical sensor; and calculating, by the computingunit, at least one of: a Tracker Definition using the first image, thesecond image, an image difference, and the Golden Tracker Definition;and a Tracker Assessment Parameter using the first image, the secondimage, the Golden Tracker Definition and a Static Tracker Definition.18. The method of claim 17 further comprising calculating, by thecomputing unit, the image difference by calculating a difference in poseof the Golden Tracker between the first image and the second image usingthe first image, the second image and the Golden Tracker Definition. 19.The method of claim 17 wherein the fixed spatial relationship is anInter Tracker Mount Spatial Relationship of a characterization jigcomprising mounting locations for the tracker and Golden Tracker,wherein, when the method calculates the Tracker Definition, the methodfurther comprises calculating, by the computing unit, a Tracker BaseGeometry using the first image, the second image, the image difference,the Inter Tracker Mount Spatial Relationship, the Golden TrackerDefinition and a Golden Tracker Base Geometry.
 20. The method of anyclaim 17 wherein the optical sensor is attached to a platform and thetracker and the Golden Tracker are attached to each other.
 21. Themethod of claim 17 wherein, when the method calculates the TrackerDefinition, the method further comprises calculating, by the computingunit, a Tracker Base Geometry using the first image, the second image,the image difference, the fixed spatial relationship, the Golden TrackerDefinition and a Golden Tracker Base Geometry.
 22. The method of claim21 wherein the tracker and the Golden Tracker are attached to eachother.
 23. The method of claim 17 further comprising providing, by thecomputing unit, at least one of the Tracker Definition or the TrackerAssessment Parameter as calculated for use in a localization procedure.24. The method of claim 17 further comprising providing, by thecomputing unit, user instructions for display by a display unit.
 25. Themethod of claim 17 wherein, when the method calculates the TrackerAssessment Parameter, the method further comprises providing, by thecomputing unit, the Tracker Assessment Parameter for display by adisplay unit.
 26. The method of claim 17 wherein the tracker and theGolden Tracker are attached to a respective mounting location on acharacterization jig.
 27. A computer-implemented method comprising thesteps of receiving, by a computing unit, an image of a tracker and aGolden Tracker attached to each other from one vantage point when bothtrackers appear in a field of view of an optical sensor; andcalculating, by the computing unit, a Tracker Assessment Parameter usingthe image, a Golden Tracker Definition, a Golden Tracker Base Geometry,a Static Tracker Definition and a Static Tracker Base Geometry.